Update all projections on snapshot installation #290
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #290 +/- ##
==========================================
- Coverage 89.68% 89.67% -0.01%
==========================================
Files 21 21
Lines 3140 3187 +47
==========================================
+ Hits 2816 2858 +42
- Misses 324 329 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
dumbbell
reviewed
Aug 27, 2024
the-mikedavis
force-pushed
the
md/update-projections-on-snapshot-install
branch
2 times, most recently
from
7f5ea0e
to
083d65c
Compare
kjnilsson
reviewed
Sep 4, 2024
kjnilsson
reviewed
Sep 4, 2024
A new `fold/3` will be created in the child commit which will fold over all payloads. `fold_matching/5` is more specific in what it matches - it takes a path and a tree and finds the matching payloads.
the-mikedavis
force-pushed
the
md/update-projections-on-snapshot-install
branch
2 times, most recently
from
9df6398
to
09f2b9f
Compare
dumbbell
reviewed
Sep 4, 2024
Ra 2.14.0 expands the `snapshot_installed` callback to pass the state before the snapshot and snapshot metadata for the old state. We'll use this in the child commit to "diff" the old and new states and find updates we need to make to projections and projection tables.
When a snapshot is installed the machine state is "swapped" from the state before the snapshot to the one contained in the snapshot. This swap jumps over potentially many commands in the log which we would have used to trigger projections. When we install a snapshot we need to update the projection state but it's not as straightforward as when "restoring" projections (on machine recovery for example) because the projection tables will already exist and potentially contain records. When we detect that a snapshot has been installed we need to diff the old and new states to find which projections have changed themselves and also update their associated projection tables. This commit also introduces `khepri_pattern_tree:fold/3`: a more generic version of `fold_matching/5` which folds over all payloads in the tree rather than those which match a path. This is used in `khepri_machine` to collect all projections in the tree into a `sets:set()`.
the-mikedavis
force-pushed
the
md/update-projections-on-snapshot-install
branch
from
09f2b9f
to
7dde526
Compare
|
I squashed the |
dumbbell
approved these changes
Sep 4, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a snapshot is installed the machine state is "swapped" from the state before the snapshot to the one contained in the snapshot. This swap jumps over potentially many commands in the log which we would have used to trigger projections. When we install a snapshot we need to update the projection state but it's not as straightforward as when "restoring" projections (on machine recovery for example) because the projection tables will already exist and potentially contain records.
When we detect that a snapshot has been installed we need to diff the old and new states to find which projections have changed themselves and also update their associated projection tables.
This requires a change in Ra to expose the old machine state and version in the
snapshot_installedcallback: rabbitmq/ra#467