There are two things that I have found that may cause that, although there may be others.
One, if you are on a surface that grabs at the skids even a little bit, it will be enough to tip the Phantom unless it is rising precisely vertically.
Second, it seems that if I don't have a good position lock just before takeoff, such as if I have moved the aircraft only a foot or so, it will tend to want to tilt in the direction of where it thinks it should be. Granted, GPS isn't that precise all the time, but I am guessing that it is pretty spot on for a short time right after the coordinates are recorded. So, if the GPS reading changes a little bit from the original lock, then the Phantom will tend to want to go that way.
Lately I have taken to forcing a GPS reading (flipping the HL switch 5 times) just before takeoff to try to insure that the lock is on the current position. This may be hogwash, but then...