SWE Common JSON Encoding in Connected systems RFC

[joanma747]

The basic example for the DataRecord on page 91 includes the values of the fields within the fields, not really useful

The example for a DataRecord inside a DataArray on page 96 does have a separate values block for the entire DataArray separate from the fields, and the encoding here is for an array of objects with key/value pairs, with the keys being the names of the fields in the DataRecord. The fields are ordered, as described as an array, so this could have a far more compact encoding (just arrays)

Going to the example of a TextEncoding on page 73 of a DataStream containing DataRecords, the encoding is efficient, not using key/value pairs (but is not an array, just a long text string that you have to parse yourself using the information from the TextEncoding.

The example on page 100 does not provide actual data encoding, does provide an interestingly nested set of fields (3 on the base level, 2 of which are vectors themselves), no idea how they propose to flatten this

[KathiSchleidt]

If it is essential to maintain the current encoding for a DataRecord with key/value pairs, would it be possible to provide an additional encoding with pure arrays?

In place of the following encoding:
"values": [ {"t": 12, "r": 3.03}, {"t": 30.1, "r": 1.68}, {"t": 40.0, "r": 1.16}, {"t": 50.1, "r": 0.85}, {"t": 59.8, "r": 0.62} ]

Enable a simpler encoding, e.g.
"values": [ [12, 3.03], [30.1, 1.68], [40.0, 1.16], [50.1, 0.85], [59.8, 0.62] ]

Maybe create a jsonEncoding type to describe which option used, similar to the already existing textEncoding. Alternatively have the DataRecord indicate that it is utilizing compact encoding via a dedicated attribute.

[alexrobin]

@KathiSchleidt @joanma747 We had such alternative encoding at some point in the early draft. We'll discuss adding this back in during our next telecon. I also think it's useful.

[alexrobin]

Some ideas for the encoding rules:

• If the DataRecord is itself within a DataArray, default to the nested JSON array encoding (like in @KathiSchleidt example)
• If not, default to JSON object encoding (I think we definitely need to keep the JSON object encoding option)
• Inside the JSONEncoding settings, allow to set the encoding type on a per record basis, which would override the defaults.

[KathiSchleidt]

@alexrobin I worry a bit about your cascade of subtleties guiding the encoding, would feel safer if we could just indicate the encoding utilized through an encodingType attribute. You already support textEncoding, why not introduce a jsonEncoding to clarify?

[alexrobin]

Discussed during 10/17 telecon.

Ok to add this feature in the JSON encoding. The idea would be to add a flag on the JSONEncoding object to turn the encoding of DataRecord as JSON arrays on or off.

[alexrobin]

@KathiSchleidt Yes, we decided to simplify the approach. We already have a JSONEncoding object. We will just add an attribute to indicate if DataRecords are encoded as objects or arrays.

[autermann]

How would we call this property? I though about recordsAsArrays, per default false. But wouldn't it make sense to also apply this to vectors? Just a generic compact? objectsAsArrays? recordsAsArrays and vectorsAsArrays?

[alexrobin]

Discussed during 12/12 telecon.

We agreed to use two separate flags: one for records called recordsAsArrays, one for vectors called vectorsAsArrays.

The goal is to support the common case where only vectors would be encoded as JSON arrays, while the surrounding records stays encoded as JSON object. For example:

{
  "time": "2024-05-16T14:32:47Z",
  "location": [-86.147852, 38.47159],
  "speed": 450,
  "heading": 126.5 
}