Airports are airports. Most small town airports don't have a tower - the tower isn't a factor. Small airports will show up an a sectional chart with a magenta circle around them. Basically, if you are in the circle, you are in the no fly zone. That said, if you contact the airport manager and tell him where you are and that you occasionally fly at or under 400 feet, he shouldnt have a problem with it.
That strip you see in my signature is 4.5 nautical miles from our local airport(KRNH). It is also within the 30 mile traffic circle for KMSP. I fly my P2 off of and around that strip. No problem. The airport manager knows where I am and has no problem. I also flew homebuilts off that strip for years and friends still land there occasionally.
If GA airplanes are flying around the neighborhood at 200 feet, they are in violation of FAR 1.119.
In the US, Part 91 (specifically 91.119
[4]) of the Federal Aviation Regulations controls the minimum safe altitudes by which aircraft can be operated in the
National Airspace System.
500 ft rule
An aircraft must maintain an altitude of 500 feet above the surface, except over open water or sparsely populated areas. In those cases, the aircraft may not be operated closer than 500 feet to any person, vessel, vehicle, or structure.
1000 ft rule
An aircraft must maintain an altitude of 1,000 feet above the highest obstacle within a horizontal radius of 2,000 feet of the aircraft over any congested area of a city, town, or settlement, or over any open air assembly of persons.
Other aircraft, such as helicopters, powered parachutes, and weight-shift-control aircraft, are not required to meet the FAR 91 minimums, so long as their operation is conducted without hazard to persons or property on the surface.