Welcome to PhantomPilots.com

Sign up for a weekly email of the latest drone news & information

Another FLYXXX.DAT converter

Discussion in 'Pro/Adv Discussion' started by BudWalker, Nov 3, 2015.

  1. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    I've been working on a tool to extract data from the FLYXXX.DAT files found on the internal SD card of the Phantom 3. Presently, the tool extracts GPS, IMU (gyroscope and accelerometer), barometer, velocity, and magnetometer information. A .csv file is then generated that can be viewed and manipulated by Excel.

    This tool has been replaced with a GUI version. Go here to use the GUI version.
     
    #1 BudWalker, Nov 3, 2015
    Last edited: May 14, 2016
    Steve Gee, Dome, JKDSensei and 4 others like this.
  2. Shammyh

    Joined:
    Apr 4, 2015
    Messages:
    325
    Likes Received:
    154
    Thanks for this.

    Would love an offline converter.
     
  3. Angus Macsac

    Joined:
    May 21, 2014
    Messages:
    112
    Likes Received:
    15
    Location:
    Ontario, Canada
    Nice work!!!
     
  4. Luap

    Joined:
    Jan 4, 2014
    Messages:
    430
    Likes Received:
    70
    This is great!
    Couldn't you compare a logged file from the DJI Pilot app, with extension .TXT converted to comprehensive csv at www.djilogs.com to figure out what other columns/records mean in the dat files for a same flight?
    The beta Litchi is also recording the logged files directly into csv in the tablet.
    Columns such as voltage for Battery_Cell1 Battery_Cell2 Battery_Cell3 Battery_Cell4 Battery_Cell5 Battery_Cell6 downlinkSignalQuality uplinkSignalQuality are logged with Litchi whilst the comprehensive logs the amp draw under CENTER_BATTERY.current. Unfortunately none of the converters log all these columns :(
    I started creating a profile in dashware from the dat coverter from www.mapsmadeeasy.com/log_viewer.
    It works pretty well - here is where I'm at MapsMadeEasy data profile - Dashware Gauges
    But I'd really like it to have Amp draw as well as voltage per cell and signal strength to work further on the profile.
     
  5. flyNfrank

    Joined:
    Feb 1, 2014
    Messages:
    4,209
    Likes Received:
    906
    Location:
    Indy, USA
    Very Nice Stuff right here Rowland. Keep the goodies coming!
     
  6. apsphoto

    Joined:
    Nov 18, 2013
    Messages:
    165
    Likes Received:
    58
    Location:
    Orange County, CA
    Fantastic I was thinking of doing this and writing it in python, but this works. Very cool thanks so much.

    Alan
     
  7. flyNfrank

    Joined:
    Feb 1, 2014
    Messages:
    4,209
    Likes Received:
    906
    Location:
    Indy, USA
    BudWalker likes this.
  8. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    Where are my manners. In all the last minute frenzy of deploying the converter, and it's website I forgot to mention the beta testers that suffered through this thing in it's less than beta state. Special thanks to go to @bbfpv, @BigAl07, @flyNfrank, @msinger , @n6vmo, and @Oso. Thank you.
     
    n6vmo likes this.
  9. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    I like your idea, but I need to think about it some. Maybe in a day or 2? Right now I'm cleaning code and figuring out the best way to deploy an offline version.
     
  10. Oso

    Oso

    Joined:
    May 19, 2015
    Messages:
    2,514
    Likes Received:
    1,473
    Location:
    Sacramento CA
    Funny you mentioned it. I am currently uploading another file from earlier today. 27% done, but its like watching water boil!
     
  11. flyNfrank

    Joined:
    Feb 1, 2014
    Messages:
    4,209
    Likes Received:
    906
    Location:
    Indy, USA

    @BudWalker, thank you for all that you do with this. Much appreciated!
     
    BigAl07 likes this.
  12. BigAl07

    BigAl07 Moderator
    Staff Member

    Joined:
    Jan 5, 2015
    Messages:
    2,362
    Likes Received:
    1,265
    Location:
    Western North Carolina
    Ditto! Thank you @BudWalker
     
  13. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    Just some status info
    1) I removed the FlyDump.jar file. I'm in the process of installing an SSL certificate. When that certificate is in place the FlyDump.jar will be put back. If you need a copy of this file before that time PM me and I'll DropBox you a copy.
    2) As mentioned above I'm installing an SSL certificate. I had not anticipated when the website was put together that an offline version would be made available. Distributing an executable that is run locally requires more security. The website may have temporary outages in the next few days. I'll try not to step on anybody's toes.
    3) The offline version is in the works. Just a bit more effort than I thought.
     
  14. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    As I mentioned I like your idea. Finding data that already exists in the tablet apps isn't my main focus, but finding some of these will probably lead to finding the stuff that doesn't make it to the tablet.

    There are a couple of ways to go about this. We could try to eyeball spreadsheet values and make the connection. That's not a lot of fun. I've done a lot hacking together special purpose programs to do the the searching for me.

    Before reverse engineering the .DAT file I had reverse engineered most of the DJI Go App .txt file format looking for the magnetometer data that I was interested in. That code is stale and I'll need to fix it before proceeding. The 4 voltage values are encoded as 2 byte integer shorts expressed as millivolts. There is also something I'm pretty sure is current (it immediately precedes the voltages), again a 2 byte integer short expressed as milliamps.

    Let me suggest we do this. My main focus is getting enough of the tools and code distributed so that others could do this if they don't want to wait on me. A lower priority for me will be to resurrect the .txt file format code and have a go at locating those values.

    BTW, something I'm keen on is to locate the propeller speed values. My plan is to start the bird, and with the right control stick input forward, right, backward, and left motions for 10 seconds each. Then I'll look at the .DAT file with the FlyDump program to find something that corresponds to those inputs.
     
  15. Luap

    Joined:
    Jan 4, 2014
    Messages:
    430
    Likes Received:
    70
    I'd be more than willing to help! I've been using a program called BinToAscii but didn't figure anything out yet - still trying though.
     
  16. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    Great! But, BinTo Ascii, and tools like it, won't help much because the payload in the .DAT is encoded. You'll need to use the FlyDump.jar that's talked about in the tech section of the website. I've taken FlyDump.jar off the website until I can get the SSL certificate working. But, you can get that FlyDump.jar here Dropbox - FlyDump.jar . Why don't you have a look. You'll probably have some questions.
     
  17. flyNfrank

    Joined:
    Feb 1, 2014
    Messages:
    4,209
    Likes Received:
    906
    Location:
    Indy, USA
    I'm curious what the reason for rpm on propeller speed? I've been searching for someone who can increase the speed of the P3. I prefer that the aircraft go faster in flight only. I don't need it to lift or carry more weight so propeller rpm is not a necessary method. From what I have studied, the better way to increase the flight speed would be by Pitch degrees. And it is not that increasing the amount of degrees is necessary. It appears that the adjustment needed is in the amount of time the aircraft remains in it's current full pitch angle, which is 30 degrees.

    Is this something you are able to do, BudWalker?
     
  18. BudWalker

    Joined:
    Jun 24, 2015
    Messages:
    772
    Likes Received:
    225
    Location:
    Santa Margarita, Ca
    Not sure I understand. I'm interested in getting ESC values for analysis purposes. What are you asking about what I'm able to do?
     
  19. JKDSensei

    Joined:
    Jun 24, 2015
    Messages:
    1,511
    Likes Received:
    658
    Location:
    USA
    The best site I've seen so far that works 100% is DJI Log Converter

    I wanted to add Dashware gauges to my P3 videos and at that site you can upload either a DAT file or a TXT Flight Record and the output gives you the option to download the DAT file converted to a plain text CSV file you can open in a spreadsheet OR....a fully functional FLYTREX csv file that works perfectly in Dashware.

    I was siked when I saw how well it worked.
     
  20. Luap

    Joined:
    Jan 4, 2014
    Messages:
    430
    Likes Received:
    70
    I run below from the window 7 command line prompt. The shell window displays numbers, lots of them, and after about 20 seconds stops and beeps(seems like it crashes). What am I doing wrong? Shouldn't the dump be saved to a txt file?
    java -jar FlyDump.jar -f <FLYXXX.DAT filename>