Looking at your terrain I can't tell if you had a clear LOS to the aircraft due to the trees. You won't get great range in that type of environment. Either you need to climb up above the trees so there is a clear LOS between you & the bird (and you can only go so high), or you need to find a more open environment in which to fly.
On the RTH. If you manually call a RTH from the remote you need to hold the button for a moment. You can't just press it quickly. If it accepts the command, the remote should begin beeping like it did later when your Phantom regained contact. There is always the possibility that you may have lost video & telemetry, but the remote is still connected to the aircraft. The "connect" light on the remote isn't always accurate either. Sometimes it can be red but you are still connected. If you are still connected the Phantom will just sit there awaiting your input until eventually the low battery RTH kicks in & its begins its journey home. In a total worst case where you just aren't sure, turn off the remote for 10 seconds. That way you KNOW the Phantom lost contact & will begin the journey home. It will reconnect as it comes back into range.
