You are completely right with the law as written... You could also get into some pretty serious trouble operating an aircraft while impaired for drinking a beer on your back deck while flying your drone around, but it’s up to the federal authorities to enforce these laws. I don’t think the aviation authorities are all that keen on wasting resources on hobby drone fliers unless they’re doing something to endanger other aircraft or use their drone to commit crimes.
Not that I delved to deeply into it, but from what I’ve seen most who got into trouble over it were for issues with where they discharge their firearm without sufficient reason. No one I’ve noticed got 5-20 for shooting down an aircraft. I’m sure there are many rural cases that don’t even make it onto the radar because authorities show up, say there is no firearm related issue, tell the drone pilot to fly somewhere else and call it a day.
I’ll restate what I have said earlier. You can be right all you want. But if you know what you are doing is aggravating people and continue doing it, you are likely going to end up causing some kind of confrontation that may result in the destruction of your drone. If this happens, your not very likely to receive much help from the legal system. If it happens often enough to enough people, you can expect more restrictions down the road on when, where and how you can fly your drone.