Assuming you're flying as a hobbyist (not Part 107 Commercial). In addition to the airport restrictions, the Keys have some national park restrictions nearby such as Biscayne and Everglades. Key West is difficult due to the airport, Naval station, and Dry Tortugas national park.