-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-1058583: Add support for iceberg tables to create_table and copy_into_table statements. #2162
Conversation
…iceberg_save' into jrose_snow_1058583_iceberg_save
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
1 similar comment
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
[ | ||
StructField("a", StringType()), | ||
StructField("b", IntegerType()), | ||
] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does this work for ArrayType and MapType?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After some testing it looks like there's several edge cases that break for structured types. This change is large already so I will address those edge cases in a second PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks good. Just need to make options case insensitive
"external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL", | ||
"catalog": "SNOWFLAKE", | ||
"base_location": "snowpark_python_tests", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we also support them in a case insensitive way. i.e.
iceberg_config={
"external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL",
"CATALOG": "SNOWFLAKE",
"BASE_LOCATION": "snowpark_python_tests"
}
should also work
CHANGE_TRACKING: change_tracking, | ||
} | ||
|
||
if iceberg_config is not None: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest also checking empty dict. if iceberg_config and "external_volume" in iceberg_config
.
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing |
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-1058583
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
This PR adds support for iceberg table parameters to save_as_table and saveAsTable. I also looked into copy_into_table, but that appears to work already without the
ICEBERG
keyword. I've passed the parameters through in this function as well in case it needs to create a table.The Python test environment has weak iceberg support so these tests may need to be disabled if they prove to be flaky.