Formatting a card erases the contents on it. Anytime you want to erase it, you can format it. There are two types common formatting: Quick and Full. In quick formatting only data tables are removed so that no program can reference your info whereas in full formatting each and every bit is formatted and thus data is not recoverable. Quick formatting is good enough for most of the time. Full formatting ensures that that storage is not fragmented though that doesn't matter much as no one usually runs the video files directly from SD card.
If you want to keep the video files on SD card, you don't need to format it but in this case at one point of time your card will be full with your videos. You can always save the videos on external media and delete the videos from card. It's your choice.
Video Cache on mobile device saves your video stream for later use. You can set a limit of 2GB for auto deletion. Cache is only good if you want to run videos of 720p nature on your mobile device. You can turn it off too so that no video cache will be retained. I'm not sure if retaining video cache will affect the performance to a great degree but yes if the processors are slow, you might notice a dent. I always keep cache on with 2GB limit on my old iPad
mini 2 and I didn't see any noticeable degradation on the performance.