Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core]Introduce ResolvingFileIO #4891

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

neuyilan
Copy link
Member

@neuyilan neuyilan commented Jan 13, 2025

Purpose

Due to the introduction of an external path[1], we are unable to use the warehouse's FileIO to read and write files. So we introduced ResolvingFileIO, which allows us to choose the appropriate fileio based on the prefix scheme of the path.

This job is a part of this job[2]

[1] #4751
[2] https://cwiki.apache.org/confluence/display/PAIMON/PIP-29%3A+Introduce+Table+Multi-Location++Management

Tests

Add HybridFileIOTest.java UT

API and Format

no change

Documentation

Because the external path of the file is on the table options. So we don't know the specific file write path from the options in the catalog. So we don't know the underlying write path is an object store or not.
So we need to modify the write-buffer-spillable parameter. Make it default to false

@neuyilan neuyilan closed this Jan 15, 2025
@neuyilan neuyilan reopened this Jan 15, 2025
} catch (IOException e) {
throw new UncheckedIOException(e);
}
FileIO fileIO = new ResolvingFileIO();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to modify the default value?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we need, if user use external path, then the original fileio can not work.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you should enable it through catalog config.

@neuyilan neuyilan changed the title [Core]Introduce HybyidFileIO [Core]Introduce ResolvingFileIO Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants