I have encountered this plenty of times, depending on how quickly the situation appears to be escalating determines my reaction. First off I carry with me at all times a copy of the latest FAA guidance, my UAS flight notification paperwork (if within 5 miles of an airport), my business cards, and my lawyer's business cards. Below are just a few of the situations I have encountered:
1) Was filming a walkthrough of a nature preserve, park volunteers told me it was illegal to fly there and on top of that it was illegal to take pictures. I pulled out my copy of the FAA's guidance, the local state park regulations, and the local county park regulations then asked for them to show me what regulation banned my flight. They looked through the paperwork then said something about calling a supervisor....they disappeared and did not return.
2) Was filming a house for sale and was questioned by a nearby homeowner. Handed them my business card and told them I'd send them a birds eye view of their neighborhood once the footage was complete; they went from confrontational to fascinated. I also let them watch my FPV monitor as I filmed.
3) Was filming a Christmas show after dark and when I landed turned around to find a cop standing right behind me. He never said a word to me and I decided to do the same.
4) Was inspecting a cell phone tower and had a nearby homeowner run over and ask me what I was doing flying over his property taking pictures of his dirty back yard. I explained I was hired to inspect the cell phone tower adjacent to his property and explained the camera was not looking down at all. He was still pretty irate and was going to complain to "somebody" so I offered to show him my FPV screen and let him watch me complete the task. By the end he was all calmed down and he took one of my business cards.
My list goes on and on. My favorite was the homeowner who finally figured out it was silly to harrass me when every helicopter that flies over is recording real time video at a higher quality than I ever could, and that Google Maps' satellites photograph every inch of his house at least once a year.
I have yet to have any real problems, an almost comical situation was when I was filming in a rather seedy part of town. The minute the drone got in the air, the entire street pretty much cleared out, cars and all. Not one car or person came down the street until I was all packed up and leaving.
I think it helps that I wear a bright orange reflective vest, I wear my company shirts, and approach every flight as if I have every right to be there (which as a matter of a fact I do). I have yet to have to provide my lawyer's card.