[TOOL][WIN] Offline TXT FlightRecord to CSV Converter

I can see that your LogViewer still can't read those battery data, don't you need some help with that? :p
I'm working on it. That's why I PMed you last week ;)
 
I'll look into it. If those fields don't change their position within the log with every new DJI GO version, I'll try to add them... in the future.
decrypting algorithm is not shareable
and without it, source code is irrelevant
so, sorry, answer is no
I was wondering about the file format of the DJI logfiles, too.
I found python code "decoding" dji-txt-files here: GitHub - daviesian/phantom-decoder: Decoder for Phantom 4 txt Flight Logs from DJI Go app.
The code does not contain any decryption method but just the (proprietary) interpretation of the indiviual bytes inside the txt files. The code is not applicable to latest txt-files, the format obviously changed in the meantime - my question now: is there really an encryption or is it still just an undocumented proprietary file format?

ps: I just "analyzed" some latest dji-txt-files and it seems, that above mentioned python code is (at least partly) still applicable, just the header blob increased to a size of 100 bytes at the beginning of each logfile. After that, the "frame" structure seems to be the same, at least frame type, frame size, frame end is parsed correctly. I cannot see any encryption. I will continue exploring this.
 
Last edited:
  • Like
Reactions: speleo
where...?
The logs are encrypted by DJI GO. I'm not able to help you decrypt them if that's your next question.
 
The logs are encrypted by DJI GO. I'm not able to help you decrypt them if that's your next question.
As stated above, GitHub - daviesian/phantom-decoder: Decoder for Phantom 4 txt Flight Logs from DJI Go app. contains a well (not fully) documented format description of djis logfiles - no need to decrypt them.

Note: there is a tiny (but impacting) difference between 'encryption' and a 'proprietary format'. While the first one provides in general security by design, the latter one is in best case security by obscurity and can generally be reverse-engineered.
 
A quick way to test if something is encrypted is to use a compression program on it (e.g., Zip). In general, encrypted files will not compress significantly if the encryption is effective (there are exceptions to that rule).
 
  • Like
Reactions: koko_
A quick way to test if something is encrypted is to use a compression program on it (e.g., Zip). In general, encrypted files will not compress significantly if the encryption is effective (there are exceptions to that rule).
Yes, that's a true issue. On the other hand, not-compressable data is not necessarily encrypted - a jpg image or just raw (noisy) sensor data is hardly compressable but not encrypted (there is no encryption key involved).

However - my sample logfile is just compressable (by zip) to only 96%.
 
Update: The "basic" message structure of the log files is described on the mentioned git-hub site. It seems the content of the messages itself is indeed encrypted in some way. There are several threads addressing this encryption, e.g.: Help me reverse this
 
OK, the usual disclaimer...I don't know diddly squat (to those millennials, that means I know a little but keep it under my hat) and it's not helped by DJI's smattering of data tools. Support is not their forte, but great drones are, but only for the fortunate who have no problems.

So here's my problem. I'm not a recent Phantom owner (I have a 2 but no FPV stuff) , but one of those upstarts who own a Mavic Pro.

I’m using Bud Walker’s CsvView, DatCon and Ferraript’s Txttocsvtool. I’ve been thoroughly confused by all of the data analysis options, so I need some help.

I have a boat and need to let my Mavic know where I am on a fairly constant basis as my bird circles an island fading out-of-sight or follows me around in case a battery peters out or my RC loses connection. To those that don't understand, RTH is important when out there and sun glare impedes your view of your drone or your mobile device screen. So Home Point - Me is important, since without it, your bird will fly home to parts last known, not where you are now. There are those who say use manual means to bring it to your current location, but why buy a Mavic, if it doesn't work as advertised... do have to squint at that little radar screen while out on the water in the sun and direct my errant bird to its nest...BS!

So, I've been skeptical of my Mavic's ability to respond accurately to the change in home point designation. Recently, its been proven to not be reliable, sometimes resulting in lost Mavics. I don't know about Phantoms 4.

Two folders in the referenced DropBox, one contains data from Sep 21st, 2017 06:06PM , and another from Sep 21st, 2017 06:18PM with data in a subfolder labeled 2017-09-21 18-18-31. In the first flight, there is a recorded home distance of 39.6’ at HP Me and 13.2’ HP Recorded, well outside any GPS uncertainty – see lines 1361-1364 of PhantomHelp DJIFlightRecord_2017-09-21_(18-06-09).xls spreadsheet 2 min. 32.2 sec. and 2m 32.5 s into the flight. In the second flight, a recorded home distance of 53.3’ at HP Me and 28.2’ at HP Recorded, 2m 32.6s into the flight, again well outside any GPS uncertainty. Note that, at launch, Home Point Recorded data entry declares a home distance of zero with appropriate map, voice and display messages. So the code that extracts HP Me data from the mobile device is producing the error. From what I can tell, HP to Current Location also gives an erroneous result.

Dropbox - Mavic Log DJI Go 4 v. 4.1.11

To reiterate, I need help. I believe Mavic has a big problem.

Given the problems with DJI support, we need to develop a consistent protocol. DJI will not respond in a timely fashion. Data needs to be consistent from one app to the next. We need to help ourselves.
 
OK, the usual disclaimer...I don't know diddly squat (to those millennials, that means I know a little but keep it under my hat) and it's not helped by DJI's smattering of data tools. Support is not their forte, but great drones are, but only for the fortunate who have no problems.

So here's my problem. I'm not a recent Phantom owner (I have a 2 but no FPV stuff) , but one of those upstarts who own a Mavic Pro.

I’m using Bud Walker’s CsvView, DatCon and Ferraript’s Txttocsvtool. I’ve been thoroughly confused by all of the data analysis options, so I need some help.

I have a boat and need to let my Mavic know where I am on a fairly constant basis as my bird circles an island fading out-of-sight or follows me around in case a battery peters out or my RC loses connection. To those that don't understand, RTH is important when out there and sun glare impedes your view of your drone or your mobile device screen. So Home Point - Me is important, since without it, your bird will fly home to parts last known, not where you are now. There are those who say use manual means to bring it to your current location, but why buy a Mavic, if it doesn't work as advertised... do have to squint at that little radar screen while out on the water in the sun and direct my errant bird to its nest...BS!

So, I've been skeptical of my Mavic's ability to respond accurately to the change in home point designation. Recently, its been proven to not be reliable, sometimes resulting in lost Mavics. I don't know about Phantoms 4.

Two folders in the referenced DropBox, one contains data from Sep 21st, 2017 06:06PM , and another from Sep 21st, 2017 06:18PM with data in a subfolder labeled 2017-09-21 18-18-31. In the first flight, there is a recorded home distance of 39.6’ at HP Me and 13.2’ HP Recorded, well outside any GPS uncertainty – see lines 1361-1364 of PhantomHelp DJIFlightRecord_2017-09-21_(18-06-09).xls spreadsheet 2 min. 32.2 sec. and 2m 32.5 s into the flight. In the second flight, a recorded home distance of 53.3’ at HP Me and 28.2’ at HP Recorded, 2m 32.6s into the flight, again well outside any GPS uncertainty. Note that, at launch, Home Point Recorded data entry declares a home distance of zero with appropriate map, voice and display messages. So the code that extracts HP Me data from the mobile device is producing the error. From what I can tell, HP to Current Location also gives an erroneous result.

Dropbox - Mavic Log DJI Go 4 v. 4.1.11

To reiterate, I need help. I believe Mavic has a big problem.

Given the problems with DJI support, we need to develop a consistent protocol. DJI will not respond in a timely fashion. Data needs to be consistent from one app to the next. We need to help ourselves.
I sort of followed your thread on MavicPilots. Help me understand what the problem is. I looked at just one of the flights. PhantomHelp DJIFlightRecord_2017-09-21_[18-06-09].txt which ultimately came from the DJI Go App running on the tablet. At time 150.5 a new HomePoint was recorded which was the position of the tablet. The HP jumped from the original HP
upload_2017-9-30_16-58-15.png

to the tablet position
upload_2017-9-30_16-58-36.png

Is this not what you need?

And could you please provide the .DAT that comes from the Mavic itself. All the .csv, .xlsx, etc. files don't help much. All that's really required is the .txt and the .DAT.

As I explained to you the problem with retrieving those .DAT files has been fixed now for 4 or 5 weeks. DJI has certainly made it confusing but an explanation can be found here.
How to retrieve a V3 .DAT File

If you're on FW 01.4.000 then you need the DJI Assistant 2 V1.1.6. And, in step 3 you'll want to do "DataUpload"
selectdataupload-jpg.22315
 
Thanks, Bud.

OK, so your analysis of the flight data suggests that the HP was properly recorded to reflect the current RC/Tablet position.

It didn't land there upon pressing RTH. The PhantomHelp spreadsheet more accurately records what actually happened for the two flights I discussed. This, of course was produced by PhantomHelp from the DJI .txt file.

And yes, I did read your explanation: How to retrieve a V3 .DAT File using DJI Assistant 2 v. 1.1.6. I did that before sending my MP back to DJI. I sent it to DJI at their suggestion; since it is under warranty. I don't expect it back for awhile and they refused to do the data analysis until they had my MP, even though I always upload/syc to DJI's cloud. I'm continuing to pursue educating myself for when it returns.

I've included a concatenated DJI Assistant DAT file in the DropBox folder. Took it apart using Extract and got Flight 095 (among others) taken the same day. Note quite a few logged errors in the .csv file that DatCon produced, including "can't write param to flash", whatever that means. I've uploaded it as a .txt file to get it past the file police. You'll have to change the extension back to .csv. The second .txt file is the data log TXT file for this flight.

I believe this is the flight Sep 21st, 2017 06:18PM. DJI Flight Log Viewer - PhantomHelp.com and DJI Flight Log Viewer - PhantomHelp.com

I don't think I caught the earlier flights in question. so looking for Flight 091 and 093. But you may have better luck.

Just to be clear, pressing RTH only once resulted in landing at/near the RC/iPad location (and then several yards away) for the four flights on 9/21/2017, all with the latest updated s/w and f/w. It could be a h/w problem or my missing something, but the PhantomHelp .csv files claim a disparity between HP of RC and HP Recorded.

Thanks again for your help.
 

Attachments

  • FLY095 - Copy.txt
    7 MB · Views: 615
  • DJIFlightRecord_2017-09-21_[18-18-31].txt
    347.1 KB · Views: 505
Thanks, Bud.

OK, so your analysis of the flight data suggests that the HP was properly recorded to reflect the current RC/Tablet position.

It didn't land there upon pressing RTH. The PhantomHelp spreadsheet more accurately records what actually happened for the two flights I discussed. This, of course was produced by PhantomHelp from the DJI .txt file.

And yes, I did read your explanation: How to retrieve a V3 .DAT File using DJI Assistant 2 v. 1.1.6. I did that before sending my MP back to DJI. I sent it to DJI at their suggestion; since it is under warranty. I don't expect it back for awhile and they refused to do the data analysis until they had my MP, even though I always upload/syc to DJI's cloud. I'm continuing to pursue educating myself for when it returns.

I've included a concatenated DJI Assistant DAT file in the DropBox folder. Took it apart using Extract and got Flight 095 (among others) taken the same day. Note quite a few logged errors in the .csv file that DatCon produced, including "can't write param to flash", whatever that means. I've uploaded it as a .txt file to get it past the file police. You'll have to change the extension back to .csv. The second .txt file is the data log TXT file for this flight.

I believe this is the flight Sep 21st, 2017 06:18PM. DJI Flight Log Viewer - PhantomHelp.com and DJI Flight Log Viewer - PhantomHelp.com

I don't think I caught the earlier flights in question. so looking for Flight 091 and 093. But you may have better luck.

Just to be clear, pressing RTH only once resulted in landing at/near the RC/iPad location (and then several yards away) for the four flights on 9/21/2017, all with the latest updated s/w and f/w. It could be a h/w problem or my missing something, but the PhantomHelp .csv files claim a disparity between HP of RC and HP Recorded.

Thanks again for your help.
I feel like we're hijacking this thread. After this I think we need to start our own thread if there is anymore to report. This discussion is also probably relevant to other platforms. I'll post my findings back over on @scjerry 's original thread in MavicPilots. see
Drone did not RTH!

In summary, the Mavic landed where it did and didn't travel back to to the HP because it was within 20 meters of the HP. FLY095 didn't match any of the .txt files that were accessible through to Dropbox link. However, both FLY094 and FLY095 exhibited the behavior at issue. Looking at FLY095 it can be seen from the eventLog stream that RTH was initiated at 163 secs

163.044 : 944416899 : 10226 [L-RC]rc cmd:STD_GO_HOME
163.185 : 945048350 : 10233 [L-FLYMODE][Ctrl<7>] REQ_RC_GO_HOME AUTO_LANDING ctrl_auto_landing

Since the AC was within 20 meters of the HP the response was for flightAction to enter the TOO_CLOSE_GOHOME_LANDING. Shortly after that flyCState entered the AutoLanding state
upload_2017-10-1_9-58-3.png
 
  • Like
Reactions: sar104
new version is out
BudWalker warned me that latest txt logs made by iOS's DJI GO don't contain CUSTOM records, at least Mavics' logs
so I made some modifications to my tool
since now, the tool will give only those CUSTOM fields, that really exist in logs
and I made new group called CALC - it contains values calculated by my tool - actual distance, travelled distance and running max values
also I added new field APP_SER_WARN.warn which is similar to APP_WARN.warn, but contains serious warnings
 
  • Like
Reactions: Hans 75 and Digdat0
new version is out
BudWalker warned me that latest txt logs made by iOS's DJI GO don't contain CUSTOM records, at least Mavics' logs
so I made some modifications to my tool
since now, the tool will give only those CUSTOM fields, that really exist in logs
and I made new group called CALC - it contains values calculated by my tool - actual distance, travelled distance and running max values
also I added new field APP_SER_WARN.warn which is similar to APP_WARN.warn, but contains serious warnings
cheers ferraript, looks like some good improvements, much appreciated.
 
Cheers - I thought it might be a link to an old version. (Plus I was being lazy - too much Christmas cheer) :)
 

Recent Posts

Members online

Forum statistics

Threads
143,096
Messages
1,467,621
Members
104,981
Latest member
brianklenhart