I found that wearing earbuds helps, but doesn't eliminate them. Also, if you see them coming, you can be sure your back is turned when they pass (not to be rude, just to pretend you don't know they're there).
As for people up to no good with drones (almost nobody), we do have to defend the hobby/business and be kind and educate people, inform them about things like wide-angle lenses, lack of zoom, expectations of privacy in public places, etc. etc.
But most people aren't worried about that. They want to ask the same basic questions about costs and capabilities. And yes, it does get old, so it tends to test ones patience.
Kids love it though. I have not grown tired of showing them the screen and answering their questions. Not one parent (if they're around) has complained about that and most of them have thanked me for taking my time to entertain them.
Chris