Decent is not unlimited in the USA. FAA says 400 foot altitude from take off point. It also references taking off from structures (land and mountains included) that are already above the surrounding area. In that case you are limited to 400 feet outward from take off point.
So if you take off from a 1000 foot tall building, you can go up to 1400 feet in a cone 400 feet wide around take off point. Flying within the cone doesnt seem to have an restrictions, so you fly out 300 and all the way down it seems.
The 400 ft rule and its related references to structures (which means antennas and buildings, not mountains), only applies to Part 107.
For recreational flight, the FAA says very little, because it is constrained by the FAA Modernization and Reform Act of 2012. It simply requires adherence to community-based rules, which most people take as referring to AMA safety guidelines. Those guidelines do not mention structures, probably since model aircraft are not expected to be buzzing buildings. They also only impose a 400 ft limit when within 3 miles of an airport (subject to agreement otherwise with the operator). Outside that distance, no altitude limit is mentioned, and the AMA has been quite consistent that there is no 400 ft AGL limit if operating within their safety program.