GUI Version of DatConverter

[BudWalker]

Proper conversion would be: Modulus( (Yaw value +360),360) . Thanks to Budwalker for the formula.

In dashware you'll need to create two calculators in the profile editor.
One Calculator = Yaw + 360
Another Calculator to take the Modulus. Below a screenshot of how to do it.

View attachment 47210

As you can see Dashware offers basic calculators that can sometimes help out. Especially useful if you want to convert KMH to MPH, feet to meters and so on.

I must be missing something. Instead of going to the trouble of creating not 1, but 2 calculators why not use the Loop Range feature as was suggested in [TOOL][WIN] Offline TXT FlightRecord to CSV Converter
Also, @Luap hasn't your Dashware profile used the Loop Range feature since always?

 

[Luap]

I must be missing something. Instead of going to the trouble of creating not 1, but 2 calculators why not use the Loop Range feature as was suggested in [TOOL][WIN] Offline TXT FlightRecord to CSV Converter
Also, @Luap hasn't your Dashware profile used the Loop Range feature since always?

I was replying to flyNfranks initial post to show that calculators do work with the negative side of Yaw.
Yes I use the loop Range in the gauges but question was about calculators not gauges (or so I assumed).

I'm unaware of any calculators or converters in dashware that work with the -negative side that yaw category. I believe the .csv converter is able to do it and data then displayed in it's own category.

 

[BudWalker]

if we were in competition, you would still be winning

this time I would like to give you two suggestions
1.
every time I start the DatCon, it loads the last used file, so there is (in my case) one minute of unnecessary waiting and then I can finally open new file
I know you need to load the file, so you can find all those time events (Recording Start, Stop, ...)
but I think, that in general, users won't be converting the same DAT file again and again
so maybe you could:
A) put there another button [Preload file] above the TimeAxisPanel and read DAT file after user clicks on it, or
B) at least give us the option to disable automatic load of the last used DAT file (so datFileTextField will be empty after DatCon starts)

2.
except for cursor change, I can see no other indication that converting is done
my suggestion is, that after user clicks the goButton, its caption could change to "Converting" and after the conversion, caption could change back to "GO" and text like "Conversion done" could be added to LoggingPanel

I repeat those are just my suggestions, I am not demanding anything
if you decide to implement them, I'll be glad, thanks in advance

edit:
one more thing, I just found one bug:
after the preload of the input file, or after the click on GO button, you should clear all values and then start to fill them
because now, if I do two conversions, one after another, some of the columns (for example flightTime, vpsHeight, errorStatus) in the first few rows of the CSV contain old values

These are good ideas.

1. My usage pattern is probably different than most users. I'm usually loading the same file because I'm debugging something and need to use the same file. Lemme think of the right way to do this.

2. I'll do this.

As for the bug. This just can't be happening. If it is then there is something really wrong. DatCon doesn't "fill values" it writes them to the .csv as it processes the .DAT. Maybe the values from the two .DAT files really are the same for these columns?

 

[ferraript]

I am not java programmer, but I think that problem is, that you are "zeroing" those variables only in the static part of the class and not in the constructor
for example take a look into class Record92_3
when user converts first file, everything is ok
but after conversion, if user clicks the GO button again, variables contain the same values until first Record92_3 occurs, and by calling printCsvLine those values go into CSV file
I hope you understand now what I mean

 

[BudWalker]

I am not java programmer, but I think that problem is, that you are "zeroing" those variables only in the static part of the class and not in the constructor
for example take a look into class Record92_3
when user converts first file, everything is ok
but after conversion, if user clicks the GO button again, variables contain the same values until first Record92_3 occurs, and by calling printCsvLine those values go into CSV file
I hope you understand now what I mean

OK, I misunderstood what you were saying. And, that's exactly what's happening. That'll be fixed in the next beta.

 

[FredzMaxxUAV]

Got the Phantom 4. New Windows DJI Assistant shows two types of files: FC file and Vision Module File. You can select one or both, and then a DAT file is created. Even if you select both, there is only one DAT file.
The "Flight recorder mode" in the DJI Go app seems to be gone, although the manual says you should launch the DJI Go app for the flight recorder - but I can't find it anywhere...
There is also only one Micro USB port left on the P4...

 

[BudWalker]

Got the Phantom 4. New Windows DJI Assistant shows two types of files: FC file and Vision Module File. You can select one or both, and then a DAT file is created. Even if you select both, there is only one DAT file.
The "Flight recorder mode" in the DJI Go app seems to be gone, although the manual says you should launch the DJI Go app for the flight recorder - but I can't find it anywhere...
There is also only one Micro USB port left on the P4...

View attachment 47277

I had posted a question on the P4 forum asking about the existence of a .DAT for the P4. I'll link to your post over there.

Have you tried running a P4 .DAT through DatCon? Can you DropBox (or equivalent) a P4 .DAT so I can take a look?

Thank You.

 

[FredzMaxxUAV]

Just tried one, won't work... Will send you one now...

 

[BudWalker]

There is a new beta (version 2.2.4) that can be obtained here. It has the GUI changes requested by @ferraript and @Fredz The bug discovered by @ferraript has also been fixed.

 

[BudWalker]

There is a new beta (version 2.2.5) that can be obtained here. It has some new fields. Battery percentageCapacity, battery percentageVolts, and battery usefulTime. The fields tableLongitude and tableLatitude give the location of the tablet for flights with a Follow Me mission. This version also has gpsHealth. See https://datfile.net/fields.html for a description of these fields.

 

[ferraript]

I made some comparisions of CSVs from your tool (2.2.7) and my tool (from logs out of my P3A), here are some notes:
1. In description you write that Gimbal:Yaw is relative to the P3 - that's not true. If it were truth, it couldn't contain all values from -180 to 180 for Phantom.
2. "My" Gimbal.yaw seems to be showing correct values, "yours" not - for example, if Phantom is rotating clockwise, gimbal should be little bit to the left trying to catch up with the bird, (my logs show for example it is 3 degrees to the left, your logs show +/- the same degree for gimbal and for the bird)
3. Your calculation of percentageVolts is incorrect, percentage is not linear dependent to voltage (for example, P3A at 14.8 V, TXT log shows 47 %, your calculation shows 87 %). I suggest you to remove this column, as it is misleading (unless you come up with correct formula).
4. Your value of distanceHP is little bit off of mine. Are you sure you use the correct Home Point coordinates? It's the one in eventLog where the flightTime is 0 for the last time.
5. I think that distanceTravelled should be also computed from the last 0 flightTime. It looks like you are calculating it from the moment you have GPS coordinates available. But at that time GPS is just "looking for" satellites and if it is looking for longer time, it could give you very unaccurate GPS coordinates.
6. Could you also use Vincenty's formula for distances calculation? Explanation is on wiki site and also javascript sources are here.

Otherwise, the rest of the data is the same (or very similar) so I'm very happy with this. I think we can be both very proud of our jobs

 

[BudWalker]

I made some comparisions of CSVs from your tool (2.2.7) and my tool (from logs out of my P3A), here are some notes:
1. In description you write that Gimbal:Yaw is relative to the P3 - that's not true. If it were truth, it couldn't contain all values from -180 to 180 for Phantom.
2. "My" Gimbal.yaw seems to be showing correct values, "yours" not - for example, if Phantom is rotating clockwise, gimbal should be little bit to the left trying to catch up with the bird, (my logs show for example it is 3 degrees to the left, your logs show +/- the same degree for gimbal and for the bird)
3. Your calculation of percentageVolts is incorrect, percentage is not linear dependent to voltage (for example, P3A at 14.8 V, TXT log shows 47 %, your calculation shows 87 %). I suggest you to remove this column, as it is misleading (unless you come up with correct formula).
4. Your value of distanceHP is little bit off of mine. Are you sure you use the correct Home Point coordinates? It's the one in eventLog where the flightTime is 0 for the last time.
5. I think that distanceTravelled should be also computed from the last 0 flightTime. It looks like you are calculating it from the moment you have GPS coordinates available. But at that time GPS is just "looking for" satellites and if it is looking for longer time, it could give you very unaccurate GPS coordinates.
6. Could you also use Vincenty's formula for distances calculation? Explanation is on wiki site and also javascript sources are here.

Otherwise, the rest of the data is the same (or very similar) so I'm very happy with this. I think we can be both very proud of our jobs

2. The Gimbal stuff has been a science fair project since it was first put in DatCon. Keep hoping that I or one of the expert users can determine more about it. It comes from the record type that also gives the motor commanded speeds. Maybe I'll take it out of DatCon.
3. The percentageVolts will be changed. There is actually a field in the .DAT that is probably the same field that ends up in the .txt. The derived value was used because we didn't know at the time about the existence of the .DAT field and the derived value was useful to the dual battery mod guys. The percentageCapacity was also found in the .DAT and that will replace the derived percentageCapacity. BTW, DatCon uses a blank instead of a 0 for the battery usefulTime when the value isn't known.
4.Don't know about this one. DatCon (as of the next beta) uses a field found in one of the binary records to determine the HP coords. It should be the same as the coords extracted from the eventLog. In both methods the HP that's used is the last to be reported, independent of the flightStart marker.
5. I don't know which version you're looking at but this has been changed (maybe after the last beta) The HP is ignored and the distance traveled computation starts at the flightStart marker.
6. As a result of our PM yesterday I looked into Vincenty's formula. I found that the difference between Vincenty and Haversine is very small, less than 1.0E-4. The larger difference between the .txt file and DatCon is due to the difference in the sampling rate. I had picked 3.3 HZ (every 200 ticks) whereas the .txt sampling rate is 10 HZ. Varying the sampling rate does just what you'd expect. 200 HZ produces a longer flight than 10 HZ, especially for flights with a lot of curves. The higher freq means shorter distances between sampling points and, therefore, less "cutting of corners" and producing a shorter distance between two points. The best solution would be to set DatCon to 200 HZ since that is the most accurate. But, I'm going to set it to 10 HZ so the results will match what appears in the .txt file.

 

[ferraript]

5. As I wrote, I used version 2.2.7. And at the flightStart marker I already have few meters above 0 in the distanceTravelled field.
One more thing: I was thinking about the idea that (for best result) this distance should be calculated from the moment when bird leaves the ground. But there is so many misleading information in TXT log (height, VPS height, Ground/Sky field, ...), for example after flightStart marker, Ground/Sky field is sometimes repeatedly changing these two values, that I can't rely on it. So I decided I would let it be as it is now.

6. Yes, I'm aware of the fact, that different sample rate will affect calculated distance. Actually, there are at least 3 factors that bring difference between DAT and TXT distances:
a) As you wrote, thanks to lower datarate, TXT is cutting the corners in the curves, so it will have incorrect smaller distance.
b) In contrast to DAT log, because of bad reception there could be gaps in TXT log, so it wouldn't have all the data, thus travelled distance will be incorrect smaller again.
c) Both formulas, Haversine and Vincenty's, return 0 for two points that are very close to each other. And because DAT has 20x higher datarate than TXT, I can easily imagine a situation, where in 100 milliseconds, DAT will get 20x zero (sum will be 0 m) and TXT will get once 0.01 m. Multiplied by 600 (one minute flight) it will be 6 m. It's not much, but it's something. In result, DAT distance will be incorrect smaller. But maybe I'm just exaggerating and nothing like that can happen in reality

If you say that difference between those two formulas is so small, then it's fine (at least) for me. I was afraid that the difference will be much much larger.

I'll wait for your next converter version. See ya.

 

[Ed209]

Great work on this! I really appreaciate everyone working on this. I'm analyzing flight data to squeeze the most out of my Phantoms' on extreme distance runs. I commented on one of Luaps videos where mentioned the latest one isn't on the site. What's missing and is an update to the Dashware profile in the works? Thanks again.

 

[Luap]

Great work on this! I really appreaciate everyone working on this. I'm analyzing flight data to squeeze the most out of my Phantoms' on extreme distance runs. I commented on one of Luaps videos where mentioned the latest one isn't on the site. What's missing and is an update to the Dashware profile in the works? Thanks again.

The new Dashware version is available here. If any one has any problems/suggestions let us know!
DatFile.net/dashware

 

[ferraript]

I am cleaning up my DAT files (especially filtering out useless ones) and I just found out that I have some cases where log from one flight is divided into several DAT files
It looks like the maximum size of DAT file is 300 MiB

so my question/suggestion is, would it be possible to process all associated files at once in this case?

for example, I have one of my flight logs divided into FLY041.DAT (300 MiB), FLY042.DAT (300 MiB) and FLY043.DAT (42 MiB)
now, I have to open all three files in the converter and create all those CSVs and then join them together
but, in every CSV, column offsetTime restarts from 0 and also there are little gaps (no data or zero data), because it take some time until first record of that type occurs again

and I would like to suggest you, that if I open FLY041.DAT in converter and it finds out, that filesize is 300 MiB, it will try to open following files and process them too, until it processes the file that is smaller than 300 MiB

what do you think?

note:
of course, for me personally it is no problem to take all three DAT files and to join them into one DAT file and then process that one, that works just fine in your converter
but I think it would help other people who are not so technically mature

 

[flyNfrank]

I have tried loading a couple P4 DAT files with version 2.2.7 with no luck. It states the file is not a DAT file. I tried renaming the file which made no difference.

Do I possibly have the R/C DAT file that might be causing this? Or did I make the mistake that the DatCon was able to convert the P4 files?

 

[flyNfrank]

I am cleaning up my DAT files (especially filtering out useless ones) and I just found out that I have some cases where log from one flight is divided into several DAT files
It looks like the maximum size of DAT file is 300 MiB

so my question/suggestion is, would it be possible to process all associated files at once in this case?

for example, I have one of my flight logs divided into FLY041.DAT (300 MiB), FLY042.DAT (300 MiB) and FLY043.DAT (42 MiB)
now, I have to open all three files in the converter and create all those CSVs and then join them together
but, in every CSV, column offsetTime restarts from 0 and also there are little gaps (no data or zero data), because it take some time until first record of that type occurs again

and I would like to suggest you, that if I open FLY041.DAT in converter and it finds out, that filesize is 300 MiB, it will try to open following files and process them too, until it processes the file that is smaller than 300 MiB

what do you think?

note:
of course, for me personally it is no problem to take all three DAT files and to join them into one DAT file and then process that one, that works just fine in your converter
but I think it would help other people who are not so technically mature

The DAT setup begins recording as soon as there is power. I calibrate my compass prior to each flight and on the DAT setup it records the A/C going through the motions in the calibration. If you need to see something from the point of power up the DAT is priceless in that respect.

I also believe you have the option to manually input the point in which you want the log to begin. So if it is starting to soon to your liking, just plug in the start point you prefer. That is a lot less complicated then rebuilding the software.

 

[BudWalker]

I have tried loading a couple P4 DAT files with version 2.2.7 with no luck. It states the file is not a DAT file. I tried renaming the file which made no difference.

Do I possibly have the R/C DAT file that might be causing this? Or did I make the mistake that the DatCon was able to convert the P4 files?

DatCon isn't able to convert P4 .DAT files.

 

[flyNfrank]

DatCon isn't able to convert P4 .DAT files.

Thanks for the info!