Improve error margin handling for FlySight v1 and v2 devices

[pr3d4t0r]

Initial Discord/FlySight discussion

Does FlySight viewer, Sky Derby, or or any of the other calculation software use a Decimal library to handle precision and rounding mode instead of float values? Given a large enough data sample or a fast enough sampling rate, some of the "skipped" rows in a dataset occured because the startTime = timeUnix-baseTime; startTime + 3.0 ends up in 2.999999 or 3.000011123342 or whatever. The errors/skipping occurs because of precision and rounding in timeUnix when the time stamps were recorded (there are ~300 samples ±50 for the FlySights set to a 100 ms sample rate).
That's why I was seeing occasional "missing rows" or rows with no data. Please let me know how you guys solved this. So far I'm not using Decimal or np.round() and focused only on standard library round(dataSeries, decimals = 2).
@aleksandr Kunin - ⬆️👽🙏🏻
In the ISC speed calculation I figure out the sampling rate like this:
step = data.plotTime.diff().dropna().mode().iloc[0]
step is not 0.1 with a 100 ms rate. It's 0.99999994269999 or whatever.

Aleksandr : I am using Decimal type everywhere - in Ruby and in database.
You know, people are extremely picky when it comes to their personal scores 🙂
In JavaScript/TypeScript however there is no built-in type for this, so I'd try to utilize floats.

Michael: In FlySight VIewer, I just use an error margin of half the time accuracy for this, so I'm looking for records from t to t + 3.0005 s. Actually, looking at it now, it seems as though this needs to be updated to 0.0005 s, as it's currently 0.005 s (FlySight used to store time to 2 decimal places only). The motivation for this figure is that it's not possible to distinguish finer time increments in the file itself, so any difference smaller than this must be an error in floating point representation.

pr3d: Excellent, noted and thank you. I think it would make sense to configure the error margin based on the FlySight version. Something like:

power = -4.0 if device.version == FlySightVersion. else -3.0
errorMargin = 5.0*(10**power)

Does the FlySight report the max time accuracy somewhere, or is that a known value from the microcontroller, or how? If it’s reported somewhere we could use that instead of hardcoding 5.0. Please let me know what you think.

Michael: Time accuracy is actually much better than the number of decimal places written to the log file--typically on the order of 100 ns. For the error margin, I would use half the least significant digit in the log file. So if you're looking at an old log file with two decimal places it would be 0.005 s, but for newer files with three decimal places it should be 0.0005 s.

pr3d: This means I'll have to grab a few rows of the CSV as text, check the number of digits to the right of the decimal point. I guess I can do this during FlySight CSV file type identification, or at least implement a function that does just that.
I may have to define a SpeedRun class that encapsulates all the metadata and heuristics. That would open the door to serializing a processed file to a document object and use a database for future processing instead of reading the data lake every time something needs to be calculated.