A lot of great advice here. I've heard it said that peoples' property rights end at 83' AGL - don't know if that's true and really don't want to press the issue.
I think droning is a lot like cell phone etiquette. It's perfectly legal to be loud and cuss on a city bus, but it brings on a whole host of unwanted issues and attention. People fear the unknown (which is a handy survival mechanism), so if you stop over their house, they fear the worst. I usually fly around 200' in my neighborhood, but I'm always on the way somewhere. So, a neighbor *may* hear me flying over, but it's nothing more than a passing curiosity. Now if I was 100', or very slow or stopped, they infer that I'm taking pics or video of them. And fairly so.
I find it easy to both fly the way I want and to attract as little attention as possible. "Security through obscurity" is a good mantra here as well. The only time my front LEDs are on is when returning at night; otherwise I prefer to keep a low profile. I like to keep some distance between me and whatever landmark/building I'm flying around so it's easier to remain anonymous; conversely, when I'm flying back, I don't fly *straight* back at myself at a million feet and then descend straight down like a searchlight.
Last night, I did some flying around a prominent civic building in the area, and it was mostly deserted, but I still kept a low profile by flying 500-1000 feet away. Everything I did was perfectly legal and safe, but that doesn't mean that jumpy people wouldn't be alarmed and try to cause trouble. I've never had any issues at all.