I went out this morning with a different battery. The problem is much better, but it is still there. I tried a Litchi flight, and that wasn't any better, so I think the issue is in the drone itself.
Also, for some reason, the litchi video file is corrupt, it won't even let me copy it off...