It is a USB Android issue. I have had a few things go wrong and figured them out. Here are my findings.
1) It does not matter much when you turn things on or off. But I usually have the bird and remote on before I plug in the USB cable to my phone. I have also turned them off and on again while phone was still connected without much issue.
2) You need to plug in the USB then slide down the top Android window shade to select anything other than "USB Charging". That tells the phone you want to use the USB for communication.
3) If you ever have an issue with it not connecting, defaulting to DJI GO when you want to use Litchi or vice versa, this is what you need to do, at least on my Samsung S7. Go to Settings/Device tab/Applications/Application Manager, then go to ALL the apps that use USB, especially DJI GO and Litchi, click on each app and under "Set as Default" make sure to "Clear Defaults" if it is enabled. This way when you plug in USB, then select "transfer images", "connect as Midi device" etc. it will ask you what app you want to use with the device (your remote) that was just connected to the USB port. You then can select DJI GO, Litchi, etc. but be sure to deselect the "Use this App by Default" or similar message. I had trouble connecting my phone to my remote for over 1/2 hour once. I finally figured out that some USB endoscope app was the "Default Device" I unchecked "set as Default" and I can now connect easy as pie now.
Marshmallow has these new USB issues that are a pain, but once you figure them out it's no so bad. One problem I still have is that if I have a loose USB connection like if the plug comes half way out or something, it will reboot my phone, which is a pain. I have used 4 Android phones and an iPad mini with my P3A and P3P. They all work well if you get the issues sorted out.