⇋ backpack

MessagePack implementation with back-referencing extension.

Features

BackPack implements MessagePack specification and the following extensions:

• Timestamp extension.
• Map and Set extensions.
• Back-referencing extension.

BackPack does not depend on Node- or browser-specific APIs.

Back-referencing

It's a really simple extension that instructs serializer to replace short repeating strings and property names with numeric ids (references) and keep a map of references to strings in the message header. When deserializing, it does the opposite: reads the message header and then replaces references with associated strings. This extension allows to shave off 20-30% from payload size.

Notes

• References are implemented as extension formats, specifically fixext1 and fixext2.

• There is a maximum number of references that can be safely stored: $2^{16} = 65536$. I don't think there is a point in storing 32 bit long references, since the overhead mostly makes the whole approach non-viable.

• Right now all strings that are 2-16 bytes characters long get turned into references. Turning into references only duplicated strings would make gains even more noticeable, but it's not an easy feat.

Usage

Doesn't get any simpler than this:

import { serialize, deserialize } from '@nrsk/backpack'

const data = {
  /* ... */
}

const se = serialize(data)
const de = deserialize(se)

Benchmarks

It's not fast, but not slow either, and performs mostly on par with the official JS implementation while producing smaller outputs for the same inputs. Detailed benchmarks for both size and performance can be found here.

Todo

This is mostly about public API:

• Add options:
  • Disable built-ins (default: false). If true, disables all built-in extensions;
  • Use Timestamp extension (default: true);
  • Use Reference extension (default: true);
  • Use Map extension (default: true);
  • Use Set extension (default: true);
  • Ignore undefined (default: false). If false, it will be encoded as null. Setting to true should theoretically allow one to implement a custom extension to encode undefined as a distinct format/type.
  • Throw on unknown data type (default: true). Setting to false will just ignore unknown data types and formats.
• Rewrite extension system.
• Add tests.

License

MIT.