Not worry so much about a wide ban, but more about public opinion and individual bans. Paranoia is already rampant but I have faith in humanity (Pressure cookers have not been banned yet). You will always have idiotic useless lawmakers trying to appear relevant and as if they had a purpose (Think congress representatives trying to ban French Fries a few years back). They will create controversy and that is what bothers me. It would not be fair to create laws that take all the fun out or stop legitimate business purposes.
I am all about regulations actually. As a caring, careful, considered and safe operator, I would welcome some regulations, because the grey area is actually more impeding. But they need to be considered with a clear mind and not based out of reaction to fear.
Without clear legal guidelines, it is up to anyone to interpret what is permissible or not. What if your mayor is swayed into thinking that a GoPro equipped Quadcopter's only purpose is to spy on your neighbor? City wide ban! What if your park warden saw someone flying a Phantom over people and crashed near them and decides to ban them? There is no appeal if there is no law.
What if the news outlets keep associating the current and dominating negative hot topics all together (Drone+Privacy+Terrorists+Danger). Then, you're just 1 mash-up away from people seeing you as a bad person and it makes it very hard to do anything.
I agree, RC planes, etc. have been around for a while, but we now have a lot more people doing it, as the technology is more advanced and affordable. This means a greater chance for something stupid to happen and also a lot more exposure.