I would be the same. We invest so much in these toys that the idea of losing/crashing/not being in control is truly frightening. And then your hands start to tremble which makes you tense which makes it worse.
Mind you, when I flew helis it was worse because, in general, they are much more...