Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Update possible reasons for commit conflicts #3518

Closed
wants to merge 1 commit into from
Closed

[core] Update possible reasons for commit conflicts #3518

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 9 additions & 8 deletions paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

package org.apache.paimon.operation;

import org.apache.paimon.CoreOptions;
import org.apache.paimon.Snapshot;
import org.apache.paimon.annotation.VisibleForTesting;
import org.apache.paimon.data.BinaryRow;
Expand Down Expand Up @@ -1115,12 +1116,10 @@ private Pair<RuntimeException, RuntimeException> createConflictException(
"Don't panic!",
"Conflicts during commits are normal and this failure is intended to resolve the conflicts.",
"Conflicts are mainly caused by the following scenarios:",
"1. Your job is suffering from back-pressuring.",
" There are too many snapshots waiting to be committed "
+ "and an exception occurred during the commit procedure "
+ "(most probably due to checkpoint timeout).",
" See https://paimon.apache.org/docs/master/maintenance/write-performance/ "
+ "for how to improve writing performance.",
"1. Data is written into expired partitions.",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can check delete files and know it is the true reason?

" Please check if you have set "
+ CoreOptions.PARTITION_EXPIRATION_TIME.key()
+ " and filter out expired data.",
"2. Multiple jobs are writing into the same partition at the same time, "
+ "or you use STATEMENT SET to execute multiple INSERT statements into the same Paimon table.",
" You'll probably see different base commit user and current commit user below.",
Expand All @@ -1130,7 +1129,8 @@ private Pair<RuntimeException, RuntimeException> createConflictException(
"3. You're recovering from an old savepoint, or you're creating multiple jobs from a savepoint.",
" The job will fail continuously in this scenario to protect metadata from corruption.",
" You can either recover from the latest savepoint, "
+ "or you can revert the table to the snapshot corresponding to the old savepoint.");
+ "or you can revert the table to the snapshot corresponding to the old savepoint.",
"4. The parallelism of committer, or the parallelism of compact coordinator (for append only scalable table) is not 1.");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need this if we have runtime check.

String commitUserString =
"Base commit user is: "
+ baseCommitUser
Expand Down Expand Up @@ -1162,7 +1162,8 @@ private Pair<RuntimeException, RuntimeException> createConflictException(
if (baseEntries.size() > maxEntry || changes.size() > maxEntry) {
baseEntriesString =
"Base entries are:\n"
+ baseEntries.subList(0, Math.min(baseEntries.size(), maxEntry))
+ baseEntries
.subList(0, Math.min(baseEntries.size(), maxEntry))
.stream()
.map(Object::toString)
.collect(Collectors.joining("\n"));
Expand Down
Loading