Hardware has a lot to do with it. If you have a lower spec device then try some or all of the following:
Reduce the quality of the stream (app settings);
Reboot the phone/use an app killer just before you load the Vision app - clear the memory out as much as you can;
Either turn off the cell radio or put that into aeroplane mode or at least switch off cellular data - phone won't be using resources to try and sync your emails, calendar data, etc;
Turn the phone GPS off and see if that helps - be aware it will affect the Find my Vision function because although your Phantom will know where it is no problem, you're phone won't know where it is;
Keep the flat side of the range extender pointed at the Vision as nearly as possible - use the swivel mount as you climb and descend and turn yourself to keep it facing the aircraft all the time;
Some have reported squeezing more range/a better signal by keeping a side of the camera on the aircraft orientated towards them most of the time - the wifi antennas are mounted on the two sides of the camera housing.
I have a Nexus 4 running Kitkat (no slouch of a phone, but not as fast as the very latest generation) and with the stream set to minimum quality I get very little lag indeed and on the medium setting it's about a quarter of a second at most at distance. That's without doing anything specific to clear out ram, etc, as I'm happy with what I've got.
Hope this helps.