Encoding recommendation

[martindurant]

I don't know if the spec here has a place for this, but I wonder if there are any opinions on the storage encoding for geo data.

I would expect the following to perform the best, but I have not done testing. It would be interesting if this repo could make some interesting measures of what decisions make the biggest difference.

• FLOAT64 data is probably needed to preserve accuracy on lat/lon data
• BYTE_STREAM_SPLIT will make this much more compressible, especially for the case that the data in a page has similar values
• v2 pages, to separate the poorly compressible but compact repetition values from the compressed data
• Zstd is probably the best compressor in terms of space/CPU tradeoff

For the case that somewhat lossy compression is allowable:

• FLOAT32 (of course)
• scale/offset (to be interpreted in the application layer)
• bitrounding to account for information content (needs no interpreting; https://www.nature.com/articles/s43588-021-00156-2 )

( @kylebarron suggested this might be a good discussion to have)

[kylebarron]

To clarify I assume you're referring to the native encodings, not the serialized WKB encoding, which is just a binary column?

[martindurant]

Correct. I don't think there will be too many options for WKT/WKB columns, and any compressor should pick up interior repetitions in that case. I suppose DELTA_LEN encoding could be tried, but my hunch is, that it won't matter.