[iceberg] Introduce feature to migrate table from iceberg to paimon #4639
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Please write [WIP] in PR title. |
JingsongLi
changed the title
LsomeYeah
force-pushed
the
migrate-iceberg
branch
from
7e6b49f to
898eed1
Compare
LsomeYeah
changed the title
tsreaper
reviewed
Dec 10, 2024
| this.paimonCatalog = paimonCatalog; | ||
| this.paimonFileIO = paimonCatalog.fileIO(); | ||
| this.paimonDatabaseName = paimonDatabaseName; | ||
| this.paimonTableNameame = paimonTableNameame; |
There was a problem hiding this comment.
Suggested change
| this.paimonTableNameame = paimonTableNameame; | |
| this.paimonTableName = paimonTableName; |
| Schema paimonSchema = icebergSchemaToPaimonSchema(icebergMetadata); | ||
| Identifier paimonIdentifier = Identifier.create(paimonDatabaseName, paimonTableNameame); | ||
|
|
||
| paimonCatalog.createDatabase(paimonDatabaseName, false); |
There was a problem hiding this comment.
Why false? User must migrate to a non-existing database?
|
|
||
| private List<IcebergManifestFileMeta> checkAndFilterManifestFiles( | ||
| List<IcebergManifestFileMeta> icebergManifestFileMetas) { | ||
| if (!ignoreDelete) { |
There was a problem hiding this comment.
Why do we need such an option? If there are deletion vectors in Iceberg, and user uses this option, then the resulting data will be incorrect. Incorrect data are useless to the users.
| @@ -190,6 +199,70 @@ private static Object toTypeObject(DataType dataType, int fieldId, int depth) { | |||
| } | |||
| } | |||
|
|
|||
| public DataType getDataType() { | |||
There was a problem hiding this comment.
This class already has a dataType member. Check if dataType is null, if not just return that object, otherwise calculate data type from type string.
# Conflicts: # paimon-core/src/main/java/org/apache/paimon/iceberg/metadata/IcebergDataField.java
…ain metadata path
…tead from iceberg catalog # Conflicts: # paimon-core/src/main/resources/META-INF/services/org.apache.paimon.factories.Factory
LsomeYeah
force-pushed
the
migrate-iceberg
branch
from
cfa7ae2 to
a6fdc51
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #xxx
Paimon has supported generating Iceberg compatible metadata, so that paimon tables can be consumed directly by Iceberg readers. Now paimon try to support an action or a procedure to support migrating iceberg table to paimon table.
The general implementation idea of this feature includes the following steps:
This pr supports basic ability to migrating iceberg table to paimon, including:
Procedure or action is not included in this pr.
Tests
org.apache.paimon.iceberg.IcebergMigrateTest
API and Format
Documentation