Forgive my confusion all, are you saying that ONLY the below modes are affected with a P3 4K?
The following flight modes are affected: VR with Immersive/Joystick Head Tracking active, Follow, Focus and Track
If you are not doing any of the above "types", then it will return home and not hover...