You may feel that the rules are too strict. But had you been flying within 500 meters distance . Under 499 ft in height, within VLOS, if something went wrong , you would have been able to do something about it.
If you crash you could crash on someone. Or something that will cause damage or...