I live in a condo complex and like to fly from my patio around town. One day a neighbor ran over to my unit saying I can't be flying over houses (Which surround mine). I told him I'm in compliance with FAA rules (Which govern this) and am not being dangerous or spying on anyone. I usually keep the flight path over driveways and streets for departure and approaches.
Let me start by saying I completely sympathize with where you want to fly, and think the current FAA rules are too restrictive.
That said, you are wrong that it is within rules/guidelines to fly over houses in an urban neighborhood. This is clearly prohibited in the rules. As is flying over streets with cars driving on them (don't know if you take care to only fly over streets that have only parked cars).
Also, you are required to stay a minimum of 400' away from people, unless you have given prior notice of operations and the people that will be impacted have given their consent to be within 400' of the flying sUAV.
There are other rules/guidelines you are not following.
However, the ultimate point here is what the actual consequence of violating these rules "on the edges" so to speak. So long as you're careful to fly a course that, should you have a catastrophic mechanical failure, the risk of injury to a bystander or damage to private property is non-existent, you're probably not going to get in any more trouble than a citation and a small fine -- at the worst. Police departments do not have the resources to bother harassing drone pilots over technical violations any more than they do people driving cars.
Fly stupid -- filming a 4th of July parade, flying over people, vehicles, etc. -- there's a real risk of winding up in jail, even if nothing goes wrong.
The best way to minimize risk of getting on the wrong side of the law is to always be thinking about your aircraft having a sudden catastrophic failure and dropping out of the sky like a rock -- what's down below you, and what might happen. If the answer is only "destruction of drone", you're probably okay. If the answer includes damaging anyone's property, you're taking quite a risk. If it includes possibly injuring someone, you're a complete idiot.