fix: refine start offset after downscaling #1185
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Refining the selected timestamp offset after projection scaling. Before if it saw there were multiple projection keys, it would use the earliest of the latest by slice. But if just one of the slices has not had recent activity and is far behind, we go back to that timestamp offset on each projection scaling or restart, replaying many events.
Update this to use the earliest of the latest grouped by projection key (by slice range), to still handle the case where a previous projection instance may have been falling behind. Also recheck that there are multiple projection keys, within the latest offsets by slice.
Tested and this improves things a lot. But there can still be cases where a particular slice is far behind, and with projection scaling, happened to be processed by a projection instance for a different key. Then it will still continue to go back to that timestamp offset, until that slice moves forward. Would need something else to address this — maybe updating that offset to the current projection key, so it will eventually all be for one projection instance and not need this special case of using the earliest.