After 37 years in law enforcement I can tell you that anyone that harms, or even attempts to do you or your property harm is breaking the law. Provided there are no city ordinances that prevent you from taking off from your back yard, and you are following all the rules of law with regard to drones.
As long as you aren't a threat to anyone, or harassing others with your drone, no one can force you to stop doing what you're doing.
If this character continues to harass you in this way, or harms you or your property, he can be arrested and charged with public mischief at a minimum and possibly even worse.
Bud