Skip to content

Commit

Permalink
chore: Update ASCII diagram of chunk binary encoding
Browse files Browse the repository at this point in the history
Signed-off-by: Christian Haudum <[email protected]>
  • Loading branch information
chaudum committed Dec 14, 2024
1 parent 2de6e16 commit dc8b06a
Showing 1 changed file with 51 additions and 25 deletions.
76 changes: 51 additions & 25 deletions pkg/chunkenc/README.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,54 @@
# Chunk format
# Chunk v4 format

```
| | |
| MagicNumber(4b) | version(1b) |
| | |
--------------------------------------------------
| block-1 bytes | checksum (4b) |
--------------------------------------------------
| block-2 bytes | checksum (4b) |
--------------------------------------------------
| block-n bytes | checksum (4b) |
--------------------------------------------------
| #blocks (uvarint) |
--------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| #entries(uvarint) | mint, maxt (varint) | offset, len (uvarint) | uncompressedSize (uvarint) |
------------------------------------------------------------------------------------------------
| checksum(from #blocks) |
-------------------------------------------------------------------
| metasOffset - offset to the point with #blocks |
--------------------------------------------------
// Header
+-----------------------------------+
| Magic Number (uint32, 4 bytes) |
+-----------------------------------+
| Version (1 byte) |
+-----------------------------------+
| Encoding (1 byte) |
+-----------------------------------+
// Blocks
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| block 1 (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
| ... |
+--------------------+----------------------------+
| block N (n bytes) | checksum (uint32, 4 bytes) |
+--------------------+----------------------------+
// Metas
+------------------------------------------------------------------------------------------------------------------------+
| #blocks (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| ... |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| #entries (uvarint) | minTs (uvarint) | maxTs (uvarint) | offset (uvarint) | len (uvarint) | uncompressedSize (uvarint) |
+--------------------+-----------------+-----------------+------------------+---------------+----------------------------+
| checksum (uint32, 4 bytes) |
+------------------------------------------------------------------------------------------------------------------------+
// Structured Metadata
+---------------------------------+
| #labels (uvarint) |
+---------------+-----------------+
| len (uvarint) | value (n bytes) |
+---------------+-----------------+
| checksum (uint32, 4 bytes) |
+---------------------------------+
// Footer
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Structured Metadata
+-----------------------+--------------------------+
| len (uint64, 8 bytes) | offset (uint64, 8 bytes) | // offset to Metas
+-----------------------+--------------------------+
```

0 comments on commit dc8b06a

Please sign in to comment.