Proposal: File Header

[martinthomson]

The current structure of the format just includes records. But interpreting those records without knowing what the maximum capacity of the table is means that you can't test overflow of the table. Or, as happens often, the table does overflow and you get errors if you set the capacity poorly.

Also, I've found that the acknowledgment discipline can affect how files are interpreted. Some encodings are produced with an encoder that assumes every header is acknowledged. Others don't assume any acknowledgments. Some decoders might not care, but I have a concurrent one that needs to handle files from @afrind that put the table updates first, and assume acknowledgments between each header block. If I had a clear signal that every header block was acknowledged immediately, I could put in a synchronization point before each header block.

So, I recommend adding a header containing:

1. The acknowledgment strategy. For now, we only need a single byte to encode one of two values: 0 can mean no acknowledgements (ever), and 1 can mean that every header block is acknowledged before the encoder sends the next set of table updates. The acknowledgment strategy can double as a version number.
2. The maximum table capacity for the run, as configured. (We would need table capacity update instructions to allow this capacity to be used, now that the starting value is 0.) This can probably be a 64-bit number, to keep things simple.

With this, it should be trivial to automate tests. (And we should all be automating tests.)

[afrind]

All of the information encoded in the file name would be better off in a file header.