Order keys for s3 #96
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This exposes a bug (previously filed as mapbox#90) which occurs when items with a range key are read from the DDB event stream, an md5 hash of the key is computed and the item written to S3. The issue is that the DDB event stream handler does not (and should not) do a 'describe_table' to know which key is the HASH and which is the RANGE and therefor simply generates the md5 hash of the item keys in whatever order they happen to appear in the stream event. The s3-backfill util does do a 'describe_table' and does order the keys by declaration order which DDB requires to be HASH first, RANGE second. The different ordering of the item keys will produce a distinct md5 hash value and different S3 path/keys will result in some items appearing twice in S3 effectively corrupting the incremental backups since two valid versions will be present at the same time.
|
In case it's not clear, this bug causes incremental backups to become corrupted if the order of keys from the event stream ever change or differ from the declaration order used in s3-backfill. The incremental backup directory will be left with up to two copies of the same DDB item but with different S3 keys. Any snapshots created from the corrupted incremental backup will contain both copies of items; if used to restore a DDB table, both items for the same keys will be loaded with the last one loaded being kept (which is not the same as the desired most recent). This cannot happen with HASH keyed only items, but any table-stream with a RANGE key is susceptible. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As reported in #90 there is a bug with key ordering when computing S3 keys. The bug reporter submitted a PR #91 which was never accepted. Maybe the use of ES6 features or lack of tests to demonstrate the bug was at issue.
This PR updates the existing tests to use a range key so that the issue is obvious and an ES5 compatible fix to the incremental and backfill functions that are affected.