[fix](move-memtable) immediately return error when close wait failed #44344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
|
run buildall |
|
TeamCity be ut coverage result: |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
kaijchen
added a commit
to kaijchen/doris
that referenced
this pull request
Nov 21, 2024
github-actions bot
pushed a commit
that referenced
this pull request
Nov 21, 2024
…44344) Problem Summary: #38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss. Previously we made that change hoping to tolerate minority replica failure in this step. However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure. This PR fixes the problem by return the close_wait error immediately. The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 21, 2024
…44344) Problem Summary: #38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss. Previously we made that change hoping to tolerate minority replica failure in this step. However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure. This PR fixes the problem by return the close_wait error immediately. The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
dataroaring
pushed a commit
that referenced
this pull request
Nov 21, 2024
…wait failed #44344 (#44386) Cherry-picked from #44344 Co-authored-by: Kaijie Chen <[email protected]>
kaijchen
added a commit
to kaijchen/doris
that referenced
this pull request
Nov 22, 2024
csun5285
pushed a commit
to csun5285/doris
that referenced
this pull request
Nov 22, 2024
…pache#44344) Problem Summary: apache#38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss. Previously we made that change hoping to tolerate minority replica failure in this step. However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure. This PR fixes the problem by return the close_wait error immediately. The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
yiguolei
pushed a commit
that referenced
this pull request
Nov 22, 2024
…wait failed #44344 (#44387) Cherry-picked from #44344 Co-authored-by: Kaijie Chen <[email protected]>
morningman
pushed a commit
that referenced
this pull request
Nov 25, 2024
…44344) Problem Summary: #38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss. Previously we made that change hoping to tolerate minority replica failure in this step. However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure. This PR fixes the problem by return the close_wait error immediately. The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
yiguolei
pushed a commit
that referenced
this pull request
Nov 25, 2024
…44344) Problem Summary: #38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss. Previously we made that change hoping to tolerate minority replica failure in this step. However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure. This PR fixes the problem by return the close_wait error immediately. The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
This was referenced Nov 25, 2024
dataroaring
pushed a commit
that referenced
this pull request
Dec 3, 2024
…44552) `test_writer_v2_fault_injection` did not assert Exception being thrown, which caused false positives. i.e. Code bugged but test passed. This PR fixed this problem. Run this test without #44344, it will now report errors as expected. ``` 2024-11-25 17:03:25.463 ERROR [non-concurrent-thread-1] (ScriptContext.groovy:122) - Run test_writer_v2_fault_injection in ./doris/regression-test/suites/fault_injection_p0/test_writer_v2_fault_injection.groovy failed org.opentest4j.AssertionFailedError: expected Exception 'load timed out before close waiting', actual success ==> expected: <true> but was: <false> ```
github-actions bot
pushed a commit
that referenced
this pull request
Dec 3, 2024
…44552) `test_writer_v2_fault_injection` did not assert Exception being thrown, which caused false positives. i.e. Code bugged but test passed. This PR fixed this problem. Run this test without #44344, it will now report errors as expected. ``` 2024-11-25 17:03:25.463 ERROR [non-concurrent-thread-1] (ScriptContext.groovy:122) - Run test_writer_v2_fault_injection in ./doris/regression-test/suites/fault_injection_p0/test_writer_v2_fault_injection.groovy failed org.opentest4j.AssertionFailedError: expected Exception 'load timed out before close waiting', actual success ==> expected: <true> but was: <false> ```
liaoxin01
pushed a commit
that referenced
this pull request
Dec 7, 2024
Related PR: #44344 `VTabletWriterV2::_select_streams()` is already checking if there is enough downstream BE to meet the replication requirements. `VTabletWriterV2::close()` should tolerate those non-open streams on close wait. Debug point `VTabletWriterV2._open_streams.skip_two_backends` is added along with `VTabletWriterV2._open_streams.skip_one_backend` to check this behavior.
github-actions bot
pushed a commit
that referenced
this pull request
Dec 7, 2024
Related PR: #44344 `VTabletWriterV2::_select_streams()` is already checking if there is enough downstream BE to meet the replication requirements. `VTabletWriterV2::close()` should tolerate those non-open streams on close wait. Debug point `VTabletWriterV2._open_streams.skip_two_backends` is added along with `VTabletWriterV2._open_streams.skip_one_backend` to check this behavior.
github-actions bot
pushed a commit
that referenced
this pull request
Dec 7, 2024
Related PR: #44344 `VTabletWriterV2::_select_streams()` is already checking if there is enough downstream BE to meet the replication requirements. `VTabletWriterV2::close()` should tolerate those non-open streams on close wait. Debug point `VTabletWriterV2._open_streams.skip_two_backends` is added along with `VTabletWriterV2._open_streams.skip_one_backend` to check this behavior.
kaijchen
added a commit
to kaijchen/doris
that referenced
this pull request
Dec 9, 2024
…pache#44552) `test_writer_v2_fault_injection` did not assert Exception being thrown, which caused false positives. i.e. Code bugged but test passed. This PR fixed this problem. Run this test without apache#44344, it will now report errors as expected. ``` 2024-11-25 17:03:25.463 ERROR [non-concurrent-thread-1] (ScriptContext.groovy:122) - Run test_writer_v2_fault_injection in ./doris/regression-test/suites/fault_injection_p0/test_writer_v2_fault_injection.groovy failed org.opentest4j.AssertionFailedError: expected Exception 'load timed out before close waiting', actual success ==> expected: <true> but was: <false> ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #38003
Problem Summary:
#38003 introduced a problem where the last sink node could report success even when close wait timeout, which may cause data loss.
Previously we made that change hoping to tolerate minority replica failure in this step.
However, it turns out the last sink node could miss tablet reports from downstreams in case of close wait failure.
This PR fixes the problem by return the close_wait error immediately.
The most common error in close wait is timeout, and it should not be fault tolerant on a replica basis anyways.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)