-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core] Support branch batch/streaming read and write #2748
Conversation
paimon-core/src/main/java/org/apache/paimon/utils/BranchManager.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/AppendOnlyFileStore.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/AppendOnlyFileStore.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/KeyValueFileStore.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/KeyValueFileStore.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/operation/FileStoreCommitImpl.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/table/DataTable.java
Outdated
Show resolved
Hide resolved
paimon-core/src/test/java/org/apache/paimon/table/AppendOnlyFileStoreTableTest.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/AbstractFileStore.java
Outdated
Show resolved
Hide resolved
…ubator-paimon into branch_read_and_write
Hi @TaoZex, I browsed through your request and thx for ur work. In my humble opinion, this PR contains lots of changes and can be separated into several PRs in order to improve the efficiency of code review. For example, we can have one PR for changes in |
paimon-core/src/test/java/org/apache/paimon/table/FileStoreTableTestBase.java
Show resolved
Hide resolved
Thank you for your suggestion. We have discussed related issues in #2608. In order to be compatible with the original design and reduce changes, the form of changes is similar to the following figure: |
Timeout in ci, please rerun it. Thanks. |
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/utils/SnapshotManager.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/operation/KeyValueFileStoreScan.java
Outdated
Show resolved
Hide resolved
paimon-core/src/main/java/org/apache/paimon/schema/TableSchema.java
Outdated
Show resolved
Hide resolved
paimon-core/src/test/java/org/apache/paimon/table/FileStoreTableTestBase.java
Outdated
Show resolved
Hide resolved
paimon-core/src/test/java/org/apache/paimon/table/AppendOnlyFileStoreTableTest.java
Show resolved
Hide resolved
+1 for this, thanks @TaoZex for contribution |
Thanks for every reviewer. Happy New Year! @FangYongs @schnappi17 @xiangyuf |
@TaoZex Great work, thx for contribution. |
Purpose
Write: After data is written, the new snapshot is written to the corresponding branch directory
Read: Read the snapshot in the corresponding branch directory to get the data file position
Linked issue:
#2727
#2726
subtask of #1795
Tests
API and Format
Next
support merge branch