I've had my drone for a few months now. I always go to the park or to the beach during vacation to fly. We recently had a snowstorm so I decided to shoot some vids around my house and neighborhood at around 100' or so.
Two weeks later, my neighbor across the street flew his drone as I was washing my car. He's a car guy too so I understand why he would hover his drone on top of me and around my house and flew by my street a few times. I guess is he was about 80-90' high. And honestly, it was annoying. I know he can't see my face or anything at that height because I have a drone myself and as a serious photography nut, I know what it's capable of shooting.
However, the buzzing sound of our drone is indeed very load and annoying. I could understand why non-owners and elderly people can get paranoid about it.
I think flying at least 200-300' around residential areas would be ideal just to give courtesy to others.