Skip to content
This repository has been archived by the owner on Dec 17, 2018. It is now read-only.

Implement log compaction #36

Open
4 of 9 tasks
allengeorge opened this issue Jan 26, 2014 · 0 comments
Open
4 of 9 tasks

Implement log compaction #36

allengeorge opened this issue Jan 26, 2014 · 0 comments
Assignees
Milestone

Comments

@allengeorge
Copy link
Owner

The libraft log can grow without bound. Log compaction (i.e. snapshots) are necessary to address this. With compaction the current state is dumped to disk and log entries subsumed by this state are removed.

There are two high-level options here:

  • Snapshots are triggered without coordination on each server. Logs and snapshots may be different on different servers. Leaders have to send snapshots and followers have to process them. Snapshots have to be loaded on startup to prime the caller's state.
  • Leader triggers snapshot for the cluster. This maintains a high-degree of log coherency, but I don't think it adds much value. It requires coordination, and you still need all the messages and logic required for the first option.

Tasks

  • Take snapshots.
  • Load a mix of snapshots and log entries on startup.
  • Automatically trigger taking a snapshot.
  • Send snapshot to followers (RPCSender API, ordering, tests, etc.).
  • Receive snapshot from leader (RPCReciever API, ordering, tests, etc.).
  • Store new snapshot
  • Truncate log entries on receiving snapshot.
  • Clean up stale snapshots.
  • Truncate log entries on taking snapshot.
@ghost ghost assigned allengeorge Jan 26, 2014
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant