-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Core]Introduce ResolvingFileIO #4891
base: master
Are you sure you want to change the base?
Conversation
paimon-common/src/main/java/org/apache/paimon/fs/HybridFileIO.java
Outdated
Show resolved
Hide resolved
paimon-common/src/main/java/org/apache/paimon/fs/HybridFileIO.java
Outdated
Show resolved
Hide resolved
paimon-common/src/main/java/org/apache/paimon/fs/HybridFileIO.java
Outdated
Show resolved
Hide resolved
paimon-common/src/main/java/org/apache/paimon/fs/HybridFileIO.java
Outdated
Show resolved
Hide resolved
paimon-common/src/test/java/org/apache/paimon/fs/HybridFileIOTest.java
Outdated
Show resolved
Hide resolved
paimon-common/src/main/java/org/apache/paimon/fs/HybridFileIO.java
Outdated
Show resolved
Hide resolved
} catch (IOException e) { | ||
throw new UncheckedIOException(e); | ||
} | ||
FileIO fileIO = new ResolvingFileIO(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to modify the default value?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need, if user use external path, then the original fileio can not work.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should enable it through catalog config.
Purpose
Due to the introduction of an external path[1], we are unable to use the warehouse's
FileIO
to read and write files. So we introducedResolvingFileIO
, which allows us to choose the appropriate fileio based on the prefix scheme of the path.This job is a part of this job[2]
[1] #4751
[2] https://cwiki.apache.org/confluence/display/PAIMON/PIP-29%3A+Introduce+Table+Multi-Location++Management
Tests
Add HybridFileIOTest.java UT
API and Format
no change
Documentation
Because the external path of the file is on the table options. So we don't know the specific file write path from the options in the catalog. So we don't know the underlying write path is an object store or not.
So we need to modify the
write-buffer-spillable
parameter. Make it default to false