Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SNOW-1373556: Implement dropna using a new expression #1748

Merged
merged 3 commits into from
Jun 10, 2024
Merged

SNOW-1373556: Implement dropna using a new expression #1748

merged 3 commits into from
Jun 10, 2024

Conversation

sfc-gh-jdu
Copy link
Collaborator

@sfc-gh-jdu sfc-gh-jdu commented Jun 7, 2024 •
edited
Loading

  1. Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.

    Fixes SNOW-1373556

  2. Fill out the following pre-review checklist:

    • I am adding a new automated test(s) to verify correctness of my new code
      • If this test skips Local Testing mode, I'm requesting review from @snowflakedb/local-testing
    • I am adding new logging messages
    • I am adding a new telemetry message
    • I am adding new credentials
    • I am adding a new dependency
    • If this is a new feature/behavior, I'm adding the Local Testing parity changes.

  3. Please describe how your code solves the related issue.

    Previous implementation will fail due to maximum recursion exceeded when there are a large number of columns

@sfc-gh-jdu sfc-gh-jdu requested a review from sfc-gh-yixie June 7, 2024 22:37
@sfc-gh-jdu sfc-gh-jdu requested a review from a team as a code owner June 7, 2024 22:37
@github-actions github-actions bot added the local testing Local Testing issues/PRs label Jun 7, 2024
@sfc-gh-stan
Copy link
Collaborator

Question: why would the previous implementation fail with maximum recursion depth error?

sfc-gh-yixie
sfc-gh-yixie reviewed Jun 7, 2024
View reviewed changes
tests/integ/scala/test_dataframe_suite.py Outdated
@@ -2934,6 +2934,14 @@ def test_dropna(session, local_testing_mode):
assert "how ('bad') should be 'any' or 'all'" in str(exc_info)


def test_dropna_large_num_of_columns(session):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add @pytest.mark.localtest like below?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@sfc-gh-jdu
Copy link
Collaborator Author

Question: why would the previous implementation fail with maximum recursion depth error?

snowpark-python/src/snowflake/snowpark/_internal/analyzer/analyzer.py

Line 168 in 6a1bd53

def analyze(
analyze function is called recursively more than 1000 times because the expression is like (c1 + (c2 + (c3 + (...)))) previously

@sfc-gh-jdu sfc-gh-jdu force-pushed the jdu-SNOW-1373556-dropna branch from b05db49 to d741fc0 Compare June 7, 2024 23:58
@sfc-gh-jdu sfc-gh-jdu requested review from a team and sfc-gh-jrose June 7, 2024 23:59
@sfc-gh-yixie sfc-gh-yixie merged commit 23db87b into main Jun 10, 2024
34 checks passed
@sfc-gh-yixie sfc-gh-yixie deleted the jdu-SNOW-1373556-dropna branch June 10, 2024 16:15
@github-actions github-actions bot locked and limited conversation to collaborators Jun 10, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@sfc-gh-aalam sfc-gh-aalam sfc-gh-aalam approved these changes

@sfc-gh-yixie sfc-gh-yixie sfc-gh-yixie approved these changes

@sfc-gh-stan sfc-gh-stan Awaiting requested review from sfc-gh-stan sfc-gh-stan is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-yuwang sfc-gh-yuwang Awaiting requested review from sfc-gh-yuwang sfc-gh-yuwang is a code owner automatically assigned from snowflakedb/snowpark-python-api-reviewers

@sfc-gh-jrose sfc-gh-jrose Awaiting requested review from sfc-gh-jrose

Assignees
No one assigned
Labels
local testing Local Testing issues/PRs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sfc-gh-jdu @sfc-gh-stan @sfc-gh-yixie @sfc-gh-aalam