I would definitely re calibrate whenever my home spot has changed more than a few feet. It should only take a minute and you can never know when you might need that little insurance policy. Alps, one spot may have interferance, while another close by doesnt. It hould tell uou if calibration fails in that case, but ifylu didnt do the dance, you wouldnt know that movng it made any difference.
I'd say the drift is surely not normal, so something's going on. If the drift is always in the same direction, make sure the blades are all good. Maybe try the second set on, just to rule that out. On the cheap little quad I bought to practice on before getting the phantom, a little issue with one blade definitely made it drift a lot. Other than that, and the possibility of something really being wrong in the hardware or software, I can't think of anything obvious that would cause the drift! Both switches were in the upper most position right? Good luck resolving this, and keep us posted in case others have the same issue.
Rich