Rule of thumb:
Recalibrate whenever anything changes on your Phantom, or in its environment. If you always fly at the same sight, calibrate once at the site, and then occasionally thereafter. If you fly at different sites, recalibrate whenever you fly at a site that's different to the last one.
You'll know if you need to do a compass calibration in that the Phantom won't hover in one position; it'll persistently drift away in one particular direction.