Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1058583: Add support for iceberg tables to create_table and copy_into_table statements. #2162

Merged
merged 27 commits into from
Sep 4, 2024

Conversation

sfc-gh-jrose
Copy link
Contributor

@sfc-gh-jrose sfc-gh-jrose commented Aug 23, 2024

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1058583

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.
  3. Please describe how your code solves the related issue.

    This PR adds support for iceberg table parameters to save_as_table and saveAsTable. I also looked into copy_into_table, but that appears to work already without the ICEBERG keyword. I've passed the parameters through in this function as well in case it needs to create a table.
    The Python test environment has weak iceberg support so these tests may need to be disabled if they prove to be flaky.

@sfc-gh-jrose sfc-gh-jrose marked this pull request as ready for review August 23, 2024 23:53
@sfc-gh-jrose sfc-gh-jrose requested a review from a team as a code owner August 23, 2024 23:53
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-jrose sfc-gh-jrose changed the title SNOW-1058583: Add support for iceberg tables to create_table statements. SNOW-1058583: Add support for iceberg tables to create_table and copy_into_table statements. Aug 27, 2024
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

1 similar comment
Copy link

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-jrose sfc-gh-jrose requested a review from a team August 28, 2024 15:53
[
StructField("a", StringType()),
StructField("b", IntegerType()),
]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this work for ArrayType and MapType?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some testing it looks like there's several edge cases that break for structured types. This change is large already so I will address those edge cases in a second PR.

@sfc-gh-jrose sfc-gh-jrose requested a review from a team August 28, 2024 22:05
@snowflakedb snowflakedb deleted a comment from github-actions bot Aug 28, 2024
Copy link
Contributor

@sfc-gh-aalam sfc-gh-aalam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good. Just need to make options case insensitive

Comment on lines 260 to 262
"external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL",
"catalog": "SNOWFLAKE",
"base_location": "snowpark_python_tests",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we also support them in a case insensitive way. i.e.

iceberg_config={
            "external_volume": "PYTHON_CONNECTOR_ICEBERG_EXVOL",
            "CATALOG": "SNOWFLAKE",
            "BASE_LOCATION": "snowpark_python_tests"
}

should also work

CHANGELOG.md Outdated Show resolved Hide resolved
CHANGE_TRACKING: change_tracking,
}

if iceberg_config is not None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest also checking empty dict. if iceberg_config and "external_volume" in iceberg_config.

src/snowflake/snowpark/dataframe.py Outdated Show resolved Hide resolved
Copy link

github-actions bot commented Sep 3, 2024

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

Copy link

github-actions bot commented Sep 3, 2024

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

Copy link

github-actions bot commented Sep 3, 2024

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

Copy link

github-actions bot commented Sep 3, 2024

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

Copy link

github-actions bot commented Sep 4, 2024

Seems like your changes contain some Local Testing changes, please request review from @snowflakedb/local-testing

@sfc-gh-jrose sfc-gh-jrose merged commit dd7b0c9 into main Sep 4, 2024
34 checks passed
@sfc-gh-jrose sfc-gh-jrose deleted the jrose_snow_1058583_iceberg_save branch September 4, 2024 23:24
@github-actions github-actions bot locked and limited conversation to collaborators Sep 4, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants