I live in a forest and take off and land manually in P mode, at the end of a RTH I take over control and hand catch due to the nature of the ground. Not sure what you mean about not trusting the GPS, it's the only thing saving me, along with the barometer of course. Do you land in atti mode?
I land and hand catch in ATTI on the P4P. With the sensors on I've found it will brake, re-elevate, suddenly stop, drop, bounce around a bit, re-correct, etc when it gets close to the ground. Sometimes I land on my hard case and GPS hates trying that. ATTI goes down smooth, no bouncing.