Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 SECRET with SCOPE not recognized by iceberg_scan #80

Open
erbou opened this issue Nov 12, 2024 · 0 comments
Open

S3 SECRET with SCOPE not recognized by iceberg_scan #80

erbou opened this issue Nov 12, 2024 · 0 comments

Comments

@erbou
Copy link

erbou commented Nov 12, 2024

I am creating in-memory secrets to read Iceberg tables on S3 using DuckDB v1.1.3 (Python 3.10.11).

CREATE SECRET secret1 (
    TYPE S3,
    KEY_ID 'the_key_id',
    SECRET 'the_secret',
    ENDPOINT 'the.end.point',
    URL_STYLE 'path',
    USE_SSL true,
    SCOPE 's3://mybucket'
)

When running the query:

FROM which_secret('s3://mybucket/my-iceberg-table/metadata/xxxx-...-xxxx.metadata.json', 's3')

It returns secret1 as expected.

However, when running:

SELECT COUNT(*) FROM iceberg_scan('s3://mybucket/my-iceberg-table/metadata/xxxx-...-xxxx.metadata.json')

It fails with the error:
HTTP Error: Unable to connect to URL "https://mybucket.s3.amazonaws.com/my-iceberg-table/metadata/snap-xxxx-...-xxxx.avro": 404 (Not Found).

Interestingly, the query works when I omit the SCOPE in the secret.

Note: I am not using AWS S3 but rather an S3-compatible service provided by our cloud provider (ceph or minio).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant