Skip to content

Binary Data Format

Jump to bottom
Brian Miles edited this page Oct 17, 2024 · 1 revision

Introduction

NMEA2000 packets are binary encoded, but include a variety of different elements (such as source and destination addresses) that are not strictly relevant for long-term storage. The firmware extracts the information in the received packets, and transcodes them to binary format for storage. Hopefully, this reduces the amount of data that needs to be stored on the SD card, and therefore improves the durability of the logger between downloads.

NMEA0183 packets are simple ASCII, and are therefore simply encapsulated with a timestamp and saved to file. In order to avoid parsing the ASCII strings (which can be tricky in general), all strings are just saved in one encapsulation message.

The log files are managed by LogManager.cpp using the logger::Manager object. The ID numbers for the messages being recorded are specified as the enum PacketIDs in logger::Manager. Data is written by serialising to the Serialisable object in serialisation.h and then using the logger::Manager::Record() method.

Serialisation

NMEA2000 packets are reformatted using the NMEA2000 library, which interprets the packets into SI units. The code in N2kLogger.cpp then converts this into a binary buffer using the Serialisable class in serialisation.cpp, which allows elemental data types (and more sophisticated data types that are serialisation aware) to be incrementally added into the buffer. After all of the data has been serialised, the buffer can be written to file using the logger::Manager::Record() method.

The LogManager code automatically adds a version packet (ID 0) into the start of every file that's written, so that reader code can adapt to different versions of the data format. This includes the version information for the serialiser, the version information for the NMEA2000 writer, and the version information for the NMEA0183 writer.

Version 1.3 Format

Serialiser V1.3 supports core data packets that are useful for the NMEA2000 data stream associated with depth data, and a general NMEA0183 data packet. Data are written as a header:

  1. Packet ID (uint32) as defined following.
  2. Packet Data Size (uint32) in bytes. This is the length of the following data packets, not including this header.

followed by the data. Endianness is native, but is LSB on ESP32 implementations.

A number of the NMEA2000 packets below include a coded integer (usually a uint8) that is a marker for things like temperature source, GNSS receiver type, etc. These are transliterated from the source code in the NMEA2000 library, which attaches meaning through a number of enums. See the N2kTypes.h source in the library for details.

The packets included are:

  1. ID 0: Serialiser Version. This packet is automatically written into each data file as the first packet, so that the reader can switch behaviours to support version differences. Supports:

    1. major: serialiser major version (uint16)
    2. minor: serialiser minor version (uint16)
    3. n2000_major: Major version information for the NMEA2000 writer (uint16)
    4. n2000_minor: Minor version information for the NMEA2000 writer (uint16)
    5. n2000_patch: Patch/Build version information for the NMEA2000 writer (uint16)
    6. n0183_major: Major version information for the NMEA0183 writer (uint16)
    7. n0183_minor: Minor version information for the NMEA0183 writer (uint16)
    8. n0183_patch: Patch/Build version information for the NMEA0183 writer (uint16)
    9. imu_major: Major version information for the IMU writer (uint16)
    10. imu_minor: Minor version information for the IMU writer (uint16)
    11. imu_patch: Patch/Build version information for the IMU writer (uint16)

  2. ID 1: System Time. This is the GNSS-based time (hopefully) at the source (data types for data_source lookup are here). Supports:

    1. date: packet days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. data_source: system that provided the time information (uint8)

    The date, timestamp, and elapsed time here are different from the rest of the packets since they come from the information in the packet itself, rather than the reference that's synthesised by the logger. This is for consistency: the SystemTime message is the source of the synthesises reference, so anything else would be circular!

  3. ID 2: Attitude. This is the motion signal for the platform. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. yaw: yaw angle in radians (double)
    5. pitch: pitch angle in radians (double)
    6. roll: roll angle in radians (double)

  4. ID 3: Depth. This should be the indicated depth from the echosounder. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. depth: depth below transducer in metres (double)
    5. offset: depth offset to apply (positive or negative) in metres (double)
    6. range: maximum depth range transducer can support, in metres (double)

  5. ID 4: Course Over Ground/Speed Over Ground. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. courseOverGround: course over ground in radians (double)
    5. speedOverGround: speed over ground in metres per second (double)

  6. ID 5: GNSS Fix. Supports:

    1. date: packet days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. date: in-message days elapsed since 1970-01-01 (uint16)
    5. timestamp: in-message seconds since midnight (double)
    6. latitude: position latitude in degrees (double)
    7. longitude: position longitude in degrees (double)
    8. altitude: position altitude in metres (double)
    9. receiverType: type of receiver that made measurement (uint8)
    10. receiverMethod: method used by the receiver that made measurement (uint8)
    11. numSVs: number of SVs in view during measurement (uint8)
    12. horizontalDOP: horizontal dilution of precision estimate (double)
    13. positionDOP: position dilution of precision estimate (double)
    14. separation: geoid-ellipsoid separation estimate (double)
    15. numRefStations: number of reference stations used for corrections (uint8)
    16. refStationType: type of reference station corrections (uint8)
    17. refStationID: ID number for the reference station (uint16)
    18. correctionAge: seconds since corrections were generated (double)

    Note that the GNSS packet has a baked-in date/timestamp that reflects the time of validity of the positioning information. The logger also synthesises a date/timestamp, with associated elapsed time of the logger, using its current time reference. The first three components here are the synthesised time, elements 4-5 are what come from the message itself.

  7. ID 6: Environmental Data. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. tempSource: source of temperature measurement (e.g., inside, outside) (uint8)
    5. temperature: temperature in Kelvin (double)
    6. humiditySource: source of humidity measurement (e.g., inside, outside) (uint8)
    7. humidity: relative humidity in percent (double)
    8. pressure: atmospheric pressure in Pascals (double)

  8. ID 7: Temperature. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. tempSource: source of temperature measurement (e.g., inside, outside) (uint8)
    5. temperature: temperature in Kelvin (double)

  9. ID 8: Humidity. Supports:

    1. date: packet reception time in days elapsed since 1970-01-01 (uint16)
    2. timestamp: packet reception time in seconds since midnight (double)
    3. elapsed_time: millisecond count since logger boot at packet reception (uint32)
    4. humiditySource: source of humidity measurement (e.g., inside, outside) (uint8)
    5. humidity: relative humidity in percent (double)

  10. ID 9: Pressure. Supports: 1. date: packet reception time in days elapsed since 1970-01-01 (uint16) 2. timestamp: packet reception time in seconds since midnight (double) 3. elapsed_time: millisecond count since logger boot at packet reception (uint32) 4. pressureSource: source of pressure measurement (e.g., inside, outside) (uint8) 5. pressure: atmospheric pressure in Pascals (double)

  11. ID 10: NMEA0183 serial data. This is used for any string received on the NMEA0183 buses, which are not interpreted. Supports: 1. elapsed_time: millisecond count since logger boot at packet reception (uint32) 2. payload: NMEA0183 ASCII string, including CRC and single LF at the end (uint8)

  12. ID 11: Motion sensor data. This is used to log information from an MPU-6050 embedded on the logger board, raw off the sensor. No processing has been done to the data. Supports: 1. elapsed_time: millisecond count since logger boot at packet construction (uint32) 2. ax: acceleration (x-axis), m/s (float) 3. ay: acceleration (y-axis), m/s (float) 4. az: acceleration (z-axis), m/s (float) 5. gx: gyroscope angle rate (x-axis), degrees/s (float) 6. gy: gyroscope angle rate (y-axis), degrees/s (float) 7. gz: gyroscope angle rate (z-axis), degrees/s (float) 8. temp: die temperature on the motion sensor, degrees Centigrade (float)

  13. ID 12: Metadata. Each logger maintains an internal name (used for the WiFi SSID) and unique identifier, which are set in the firmware at configuration time. These are used to make the metadata record used for upload to DCDB, and are written into each logger file at the start of the file, after the version information. Supports: 1. name_len: Number of bytes to follow for the name string (uint32) 2. name: Logger name (char x name_len) 3. id_len: Number of bytes to follow for the identification string (uint32) 4. id: Logger identification string (char x id_len)

  14. ID 13: Algorithm Request. Each logger can hold a list of algorithms that the data should be passed through, in addition to the standard processing (file conversion, timestamping, GeoJSON conversion, upload). This allows loggers to inform the cloud processing scheme which algorithms, and parameters are required. One packet is issued per algorithm. Note that no checking is done for the algorithm names, or that the parameters are valid: this is up to the reader. Contents:

    1. alg_len: Number of bytes to follow for the algorithm string (unint32)
    2. algorithm: Name of algorithm (char x alg_len)
    3. param_len: Number of bytes to follow for the parameter string (uint32)
    4. parameters: string containing the parameters for the algorithm (char x param_len)

  15. ID 14: JSON Metadata. When the WIBL files are translated in the cloud into GeoJSON for upload to the IHO Data Center for Digital Bathymetry, a set of JSON-formatted metadata is added to the start of the file, automatically populated by the processing script. Much of the metadata cannot be determined by the algorithm, however, and each logger can therefore store an arbitrary JSON-formatted string to provide extra information. This packet store the string provided on configuration; note that no checking of the string's contents are done: what the configuration provides is what gets stored. Contents:

    1. elem_len: Number of bytes to follow for the metadata element (uint32)
    2. metadata_element: JSON-formatted metadata (char x elem_len)

  16. ID 15: NMEA0183 Packets. Each logger can be configured to record only certain NMEA0183 sentences in order to save space. This packet provides a way to specify one sentence recognition string that is being recorded when the file was started; one packet is issued per sentence being accepted. Contains:

    1. recog_len: Number of bytes to follow for the recognition string (uint32)
    2. recog_string: Three-letter recognition string for the NMEA0183 sentence being accepted for recording (char x recog_len)

  17. ID 16: Sensor Scales. Each logger can have a number of on-board sensors that can generate data in scaled integer values (to minimise storage space). This packet provides the means to record the scale factors required to translate these into usable units. The scales are encoded as a JSON-format string. Contains:

    1. config_len: Number of bytes to follow for the JSON document (uint32)
    2. config_string: The JSON-format scales document (char x 'config_len')

  18. ID 17: RawIMU Data. Some implementations of the WIBL logger contain an embedded 6-dof IMU. This packet encapsulates a raw measurement from the IMU. Note that this data may be scaled integers subject to scales from an ID16 (Sensor Scales) packet. Contains:

    1. elapsed: Elapsed time (milliseconds) on the logger at data capture (uint32)
    2. temp: Scaled temperature of the IMU logger's chip (int16)
    3. gx: Scaled gyro rate about the x-axis (int16)
    4. gy: Scaled gyro rate about the y-axis (int16)
    5. gz: Scaled gyro rate about the z-axis (int16)
    6. ax: Scaled acceleration in the x-axis (int16)
    7. ay: Scaled acceleration in the y-axis (int16)
    8. az: Scaled acceleration in the z-axis (int16)

  19. ID 18: Logger Setup. For each log file, the logger captures the current configuration of the logger, formatted as a JSON document, and stores it for future reference. Contains:

    1. setup_len: Number of bytes to follow for the JSON document (uint32)
    2. setup: JSON-formatted configuration document (char x 'setup_len')
Clone this wiki locally