mmmm, technically it should have done the RTH !!!
Maybe it did it because when you went to the car, you weren't facing to the P4 anymore and the bird lost signal with the controller, which is a good thing...
Next time that happens, simply press the RTH button. If you had stood there, waiting...