Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] [broker] Skip to persist cursor info if it failed by cursor closed #23615

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

poorbarcode
Copy link
Contributor

@poorbarcode poorbarcode commented Nov 20, 2024

Motivation

We encountered an issue as follows

time closing topic -> closing curosr in-progress cursor mark deleting
1 Mark the cursor as closing
2 Persistent cursor metadata into the cursor ledger 54781383
3 Start to close the cursor ledger 54781383
4 Mark the ledger 54781383 is closed in memory
5 Try to write metadata to the ledger 54781383, but the ledger was marked closed
6 Try to write metadata to ZK after failed writing BK
7 Start to write the ledger metadata of 54781383 to ZK
8 The ZK server closed the connection due to the pulsar trying to write a large value
9 Write to ZK failed due to the connection being lost
10 retries and keeps triggering a closing of the ZK connection
11 The retries will fail because the connection is unstable

I think we need at least 3 fixes

  • (What this current PR attempts to do) Do not continue executing the in-progress cursor mark deletion after the cursor is closed.
  • (I will push another PR to do it) Add a mechanism that limits the data size that will persist with ZK.
  • (I will push another PR to do it) Stop retries if such an issue occurs.

Modifications

  • Skip to persist cursor info if it failed by cursor closed
  • TODO: I will write a test tomorrow.

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: x

@poorbarcode poorbarcode added type/bug The PR fixed a bug or issue reported a bug release/3.3.3 release/3.0.8 release/4.0.1 labels Nov 20, 2024
@poorbarcode poorbarcode added this to the 4.1.0 milestone Nov 20, 2024
@poorbarcode poorbarcode self-assigned this Nov 20, 2024
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Nov 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs release/3.0.8 release/3.3.3 release/4.0.1 type/bug The PR fixed a bug or issue reported a bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant