Decoding trails

[scubajorgen]

@ryanbinns Sorry to abuse your issue list for this: the Adventurer allows to upload 'trails' to mysports.tomtom.com as GPX file, for track planning purposes. The GPX is converted to a proprietary format, which ends up on the watch as a file with ID 0x00b8nnnn. I am trying to decode the format, but I am stuck. I recognize the coordinates, but the metadata does not make sense. I would like to ask you if you can have a look at it. Please, could you send me your e-mail address to scubajorgen at gmail, so I can send you some examples.

[ryanbinns]

I'm not going to put my email on here, but you can attach files to the issue if you like. I'll have a look when I get some time, but that may not be for a while.

[scubajorgen]

Okay, I would not polute your development with this discussion. But I hereby send a number of GPX files I uploaded and the corresponding watch files (binary and a text representation). Reverse engineering shows what I found out so far...
I would kindly ask if it looks familiar to you. TIA
Kind regards
Jorgen
example.zip

[heliumir]

Those files are protobufs, can parse them with protobuf-inspector and create a .proto file based on what comes out of that.
It would be nice to decode files with ttwatch someday :D

[scubajorgen]

Brilliant!
I just found out the structure of the file and this odd 7 bit-based enoding for numbers. I was just about writing an encoder for GPX files myself. Knowing it is Google Protobuf would make life a lot easier. Thanx!!
Btw I will use the upload function in my Java App, version 0.2. See: http://blog.studioblueplanet.net/?page_id=566.
This would come in handy for ttwatch as well. I could provide the meaning of the encoded variables

[scubajorgen]

I've set up the protobuf definition file. Here it is.
The only thing is that in the serialized protobuf files generated by TomTom both fields in the root message have ID 1. I cannot get that working in the protobuf definition (fields in a message must have different IDs). I worked around this by setting byte[14]=0x0a (the 15th byte) after serialization before writing to the watch. When reading set byte[14]=0x12.

Apart from this field ID and the time stamps I manage to generate bytewise identical protobuf files from the same GPX.
Note: the time stamp (normally I assume the timestamp of uploading to TomTom) is a funny one: it counts 3.456 counts per seconds and it appears to have started at 07-03-2016 06:30 UTC. Odd sense of humor?
route.zip

[ryanbinns]

Thanks for doing the analysis of this. What functions would you like to see in ttwatch for this? Encoding? Decoding? Upload to the watch? Download from the watch? Anything else?

I haven't worked with (or even heard of...) Google Protobufs before. Is there a standard library for creating/decoding them? One that's not too big or hard to use?

[scubajorgen]

Personally, I don't need this, because I write my own app. I provide you with the info if you intend to add this upload function to ttwatch. I don't even know if watches other than the Adventurer support upload of routes.

Protobufs is from google (https://developers.google.com/protocol-buffers/). It is an binary alternative to xml and json and takes even less number of bytes to encode the information.
It works as follows:

• Create a .proto defining the message (compare what XSD is to an XML document I guess)
• Use protoc to compile the .proto file into a parser/builder in the language of choice (c++)
• Incorporate the parser/builder in your code
• Use the builder to create the serialized message (by setting field values and build)
• Use the parser to parse a serialized message to extract the field values