Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(iceberg): Adds support for read_iceberg with metadata_location to Daft-SQL #3701

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

RCHowell
Copy link
Contributor

This PR adds support for read_iceberg given a metadata location, and it adds this to Daft-SQL.

Note

This was tested manually because PyIceberg StaticTable has a bug where the filepaths in the iceberg metadata are resolved relative to the caller's current working directory rather than relative to the metadata file. This will need to be fixed in PyIceberg, but I wanted to close this out for now since it still works if the paths are right.

Future Additions

Testing

read_iceberg

Copy link

codspeed-hq bot commented Jan 17, 2025

CodSpeed Performance Report

Merging #3701 will degrade performances by 29.52%

Comparing rchowell/read_iceberg (258d9a9) with main (6b302af)

Summary

⚡ 1 improvements
❌ 1 regressions
✅ 25 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main rchowell/read_iceberg Change
test_iter_rows_first_row[100 Small Files] 193.5 ms 109 ms +77.49%
test_show[100 Small Files] 16.5 ms 23.4 ms -29.52%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant