Skip to content

Commit

Permalink
docs: introduced README
Browse files Browse the repository at this point in the history
  • Loading branch information
abdullin committed Dec 24, 2017
1 parent 324028e commit e67e230
Showing 1 changed file with 31 additions and 0 deletions.
31 changes: 31 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,37 @@ HappyPancake to run real-time aggregation and long-term data analysis
on the largest social website in Sweden. You can read more about it in
[Real-time Analytics with Go and LMDB](https://abdullin.com/bitgn/real-time-analytics/).

# Design

Cellar stores data in a very simple manner:

- LMDB database is used for keeping metadata (including user-defined);
- a single pre-allocated file is used to buffer all writes;
- when buffer fills, it is compressed, encrypted and added to the chunk list.

You can have **only one writer at a time**. This writer has two operations:

- `Append` - adds new bytes to the buffer, but doesn't flush it.
- `Checkpoint` - performs all the flushing and saves the checkpoints.

The store is optimized for throughput. You can efficiently execute
thousands of appends followed by a single call to `Checkpoint`.

Whenever a buffer is about to overflow (exceed the predefined max
size), it will be "sealed" into an immutable chunk (compressed,
encrypted and added to the chunk table) and replaced by a new buffer.

See tests in `writer_test.go` for sample usage patters (for both
writing and reading).


At any point in time multiple readers could be created, they will be
working against the last available snapshot, optionally reading into
the buffer as well. Readers have only one operation available for
them - reading sequentially within a given interval (or scanning
through the entire storage).


# License

3-clause BSD license.

0 comments on commit e67e230

Please sign in to comment.