Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Why not use the profile name when initialising the S3FileSystem class? #1207

Open
wudihero2 opened this issue Sep 25, 2024 · 6 comments
Open
Assignees

Comments

@wudihero2
Copy link

wudihero2 commented Sep 25, 2024

Question

Hi,
In #922 standardize some AWS credential names, but I am confused why not use below code to use aws profile name at pyiceberg.io.fsspec.py?

profile_session = AioSession(profile="xxx")
fs = s3fs.S3FileSystem(session=profile_session)

It supports the use of AWS profile names like `glue.profile-name' as in the following code at pyiceberg.catalog.glue.py

profile_name=PropertyUtil.get_first_property_value(properties, GLUE_PROFILE_NAME, DEPRECATED_PROFILE_NAME),

Maybe it would be a better improvement if we could use profile name or is there some concern I haven't considered?

@kevinjqliu
Copy link
Contributor

I think this is a feature gap on the S3 FileIO. It makes sense to support profile_name.
We would need to support both fsspec and pyarrow

Is this something you would like to contribute?

@wudihero2
Copy link
Author

Hello, I am interested in this, do I need to tag the person who will assign this task to me?

@kevinjqliu
Copy link
Contributor

@wudihero2 assigned to you :)

@sungwy
Copy link
Collaborator

sungwy commented Sep 26, 2024

Hi folks, I was under the impression that this was something that would need to be addressed in PyArrow S3FileSystem. Please see @HonahX 's earlier comment:

#922 (comment)

Previous issue on same topic: #570
PyArrow S3FileSystem: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

On that note, I'm in favor of keeping this issue open since this is a frequent question from the community, until we are able to find a solution by perhaps working with the Arrow community.

@kevinjqliu
Copy link
Contributor

#1104 (comment)
this thread is somewhat related

@wudihero2
Copy link
Author

Hi all, I checked the code of pyarrow and found that the profile_name parameter is not currently supported. The s3.* related parameters are indeed not suitable for supporting profile_name. It would be great if we could work with the Arrow community to find a solution! This package is great and can help me use iceberg for my current job, thank you !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants