Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[kv] Fix out of order exception after delete a not exist row #312

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

swuferhong
Copy link
Collaborator

@swuferhong swuferhong commented Jan 6, 2025

Purpose

Linked issue: #230

This pr is aims to fix the OutOfOrderSequenceException after delete a not exist row or input keys with MergeEngine set to FIRST_ROW. The reason of this problem is that the kvRecordBatch send from client may not generated any CDC logs,
for example, when client attempts to delete some non-existent keys or MergeEngine set to FIRST_ROW. In this case, the old implements will simply return but not update batchSequence, as doing so would cause a OutOfOrderSequenceException problem. Therefore, this pr will introduce a filed lastOffsetDelta in record batch used to calculate the lastOffset of the current batch as: [lastOffset = baseOffset + LastOffsetDelta] instead of [lastOffset = baseOffset + recordCount -1]. The reason for introducing this field is that there might be cases where the offset delta in batch does not match the recordCount. For example, when generating CDC logs for a kv table and sending a batch that only contains the deletion of non-existent kvs, no CDC logs would be generated. However, we need to increment the batchSequence for the corresponding writerId to make sure no {@link OutOfOrderSequenceException} will be thrown. In such a case, we would generate a logRecordBatch with a LastOffsetDelta of 1 but a recordCount of 0.

Tests

API and Format

Documentation

@swuferhong swuferhong force-pushed the delete-not-exist-key branch from 07942fe to 9998d3f Compare January 7, 2025 07:50
@swuferhong
Copy link
Collaborator Author

@wuchong @luoyuxia pr ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] OutOfOrderSequenceException may be thrown when write to primary key table
1 participant