-
Notifications
You must be signed in to change notification settings - Fork 2
Data File Parser
There is intentionally no mechanism on the logger to read and parse the data files being recorded. Memory space on the logger is limited, and therefore adding more code to translate the files on-board is not advisable. Instead, a Python script is available in the repository to read and translate the files.
This is implemented as a library file (wibl-python/wibl/core/logger_file.py) that reads the data packets and translates them into Python classes, and a simple driver script (wibl-python/wibl/cmd/parse_wibl_file.py) that shows how to drive it (this is a sub-command of the main WIBL Multitool). Data from a variety of other loggers can be converted into WIBL format using LogConvert, and then translated, examined, or uploaded to AWS for processing.
The logger_file.py script is intended to be imported by other code, and provides a library-like interface to the data file packets that abstracts the details of reading and translating the packets.
The core model is of a base class (DataPacket) from which all other packets are derived, and which also stores the timestamp information for the packet (which is common to all packets). Each known packet type is a sub-class that contains the information for the specific packet.
All packets support a name() method that reports a human-readable name for the packet (e.g., 'SystemTime', or 'Depth'), an id() method to provide their numerical recognition code, and a __str__() method to generate a string human-readable transcription of the packet (which includes the timestamp). All packets are therefore stream-able. Currently, the human-readable transcriptions convert SI units into more practical measures (e.g., temperatures in Kelvin are translated into degrees Celsius), but does not attempt to translate the code values for things like GNSS receiver type, temperature source, etc. Details of these can be found here.
Reading the file is handled by instantiating a PacketFactory object, and providing the file from which to read the data. It is assumed that the file is opened in binary format, and Very Bad Things will happen if this is not the case. Unknown packets are reported with a printed error message, and a return of None.
Use of PacketFactory is simple: call PacketFactory::has_more() to check whether more packets are available from the file, and use PacketFactory::next_packet() to get the next packet as a DataPacket sub-class object. The code returns None if there are no packets remaining, or if there is an unknown packet read from stream.
Beginning with v 1.1 of the script, logger_file.py can also be used to write WIBL files, which is a requirement for file editing in post-processing. The DataPacket base class is now an abstract base, with a payload() method to pack the data into a binary array, and an id() method to return the identification number for the packet. Each derived class must implement these, providing the ability for the standard serialise() method to write the packet to file on demand without knowing the details about the encoding of each different packet type.
Because Python doesn't readily allow overwritten constructors with different parameters, the code approximates this by having the constructor for each packet use keyword arguments. If the buffer keyword is given, the code should assume that it's constructing from a binary buffer; otherwise, the keyword arguments should specify the parameters required to set up the packet from scratch. This of course will be different for each packet, and it's up to the packet to manage error reporting for this. A SpecificationError exception derivative is provided to assist with this.
Conventionally, the "construct from a binary buffer" method should be buffer_constructor() while the "construct from keywords" method should be data_constructor().
An example of how to read and modify binary WIBL files in Python can be found in the wibl-python/wibl/cmd/edit_wibl_file.py script (another sub-command of the WIBL Multitool).
The wibl-python/wibl/cmd/parse_wibl_file.py script simply opens a file (in binary mode), instantiates a PacketFactory object for that file, and then reads packets (PacketFactory::next_packet()) until they are all parsed. Optionally, it can maintain a list of all packets observed in the file, and how many of each are observed. This script should be run through the WIBL Multitool.