-
Notifications
You must be signed in to change notification settings - Fork 117
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
SNOW-1300434: Merge Snowpark pandas back to Snowpark Python (#1389)
* SNOW-1300434: Merge Snowpark pandas back to Snowpark Python Signed-off-by: Naren Krishna <[email protected]> * forgot to add precommit changes Signed-off-by: Naren Krishna <[email protected]> * Get back to only Snowpark Python tests Signed-off-by: Naren Krishna <[email protected]> * fix tox env and add path to pandas changelog Signed-off-by: Naren Krishna <[email protected]> * remove unnecessary changes Signed-off-by: Naren Krishna <[email protected]> * remove checkprs Signed-off-by: Naren Krishna <[email protected]> * add back file to pre-commit-config Signed-off-by: Naren Krishna <[email protected]> * fix rebase issue Signed-off-by: Naren Krishna <[email protected]> * Add Snowpark pandas tests Signed-off-by: Naren Krishna <[email protected]> * add iris.csv to tests/resources Signed-off-by: Naren Krishna <[email protected]> * revert daily precommit changes Signed-off-by: Naren Krishna <[email protected]> * fix test_utils_suite to add iris.csv Signed-off-by: Naren Krishna <[email protected]> * fix Snowpark pandas test failures Signed-off-by: Naren Krishna <[email protected]> * fix lint Signed-off-by: Naren Krishna <[email protected]> * Update to pandas 2.2.1 and fix some more tests Signed-off-by: Naren Krishna <[email protected]> * fix more tests Signed-off-by: Naren Krishna <[email protected]> * update conftest Signed-off-by: Naren Krishna <[email protected]> * fix upload name and remove pandas 2.2.1 from requirements Signed-off-by: Naren Krishna <[email protected]> * fix doctest and read_json_empty error Signed-off-by: Naren Krishna <[email protected]> * address some comments Signed-off-by: Naren Krishna <[email protected]> * remove exclued files from precommit Signed-off-by: Naren Krishna <[email protected]> * xfail test_read_json_empty_file Signed-off-by: Naren Krishna <[email protected]> * restore dataframe reader Signed-off-by: Naren Krishna <[email protected]> * update tox.ini Signed-off-by: Naren Krishna <[email protected]> * fix test Signed-off-by: Naren Krishna <[email protected]> * snowpark_package_to_sproc_packages change Signed-off-by: Naren Krishna <[email protected]> * Snowpandas changes including commit 1008 Signed-off-by: Naren Krishna <[email protected]> * update query count Signed-off-by: Naren Krishna <[email protected]> * Pull in Modin dependency PR Signed-off-by: Naren Krishna <[email protected]> * add Varnika's binary op PR Signed-off-by: Naren Krishna <[email protected]> --------- Signed-off-by: Naren Krishna <[email protected]>
- Loading branch information
1 parent
ac18513
commit 8832a62
Showing
375 changed files
with
154,797 additions
and
3 deletions.
There are no files selected for viewing
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1,4 @@ | ||
| * @snowflakedb/snowpark-python-api-reviewers | ||
| /src/snowflake/snowpark/modin/ @snowflakedb/snowpandas | ||
| /tests/integ/modin/ @snowflakedb/snowpandas | ||
| /tests/unit/modin/ @snowflakedb/snowpandas |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| name: Snowpark pandas Changedoc Check | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize, labeled, unlabeled] | ||
| branches: | ||
| - pandas-main | ||
| paths: | ||
| - 'src/snowflake/snowpark/modin/**' | ||
|
|
||
| jobs: | ||
| check_pandas_change_doc: | ||
| runs-on: ubuntu-latest | ||
| if: ${{!contains(github.event.pull_request.labels.*.name, 'NO-PANDAS-CHANGEDOC-UPDATES')}} | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Ensure Snowpark pandas docs is updated | ||
| run: git diff --name-only --diff-filter=ACMRT ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep -q "docs/source/modin" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| name: Snowpark pandas Changelog Check | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize, labeled, unlabeled] | ||
| branches: | ||
| - pandas-main | ||
| paths: | ||
| - 'src/snowflake/snowpark/modin/**' | ||
|
|
||
| jobs: | ||
| check_pandas_change_log: | ||
| runs-on: ubuntu-latest | ||
| if: ${{!contains(github.event.pull_request.labels.*.name, 'NO-PANDAS-CHANGELOG-UPDATES')}} | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Ensure PANDAS_CHANGELOG.md is updated | ||
| run: git diff --name-only --diff-filter=ACMRT ${{ github.event.pull_request.base.sha }} ${{ github.sha }} | grep -wq "src/snowflake/snowpark/modin/plugin/PANDAS_CHANGELOG.md" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
77 changes: 77 additions & 0 deletions
77
ci/check_standalone_function_snowpark_pandas_telemetry_decorator.py
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,77 @@ | ||
| #!/usr/bin/env python3 | ||
| # | ||
| # Copyright (c) 2012-2024 Snowflake Computing Inc. All rights reserved. | ||
| # | ||
| import ast | ||
|
|
||
|
|
||
| class DecoratorError(Exception): | ||
| pass | ||
|
|
||
|
|
||
| def check_standalone_function_snowpark_pandas_telemetry_decorator( | ||
| target_file: str, | ||
| telemetry_decorator_name: str, | ||
| ) -> None: | ||
| """ | ||
| Check if all standalone functions in the target file have been decorated by the decorator with | ||
| name telemetry_decorator_name. | ||
| Raises a DecoratorError if the decorator is missing. | ||
| Args: | ||
| target_file (str): Path to the target file. | ||
| telemetry_decorator_name: Name of the telemetry decorator that is checked. | ||
| """ | ||
| # Get the source code of the target file | ||
| with open(target_file) as file: | ||
| source_code = file.read() | ||
| assert source_code.strip(), f"Source code in '{target_file}' is empty." | ||
| # Parse the abstract syntax tree | ||
| tree = ast.parse(source_code) | ||
|
|
||
| # List of str: function names that need the decorator. | ||
| failed_funcs = [] | ||
|
|
||
| # Apply the decorator to the functions with matching return types | ||
| # Exclude sub-functions with iter_child_nodes which yields direct child nodes | ||
| for node in ast.iter_child_nodes(tree): | ||
| if ( | ||
| isinstance(node, ast.FunctionDef) # Check if it is function type | ||
| and not node.name.startswith( | ||
| "_" | ||
| ) # the function is not private (does not start with an underscore) | ||
| and node.name | ||
| ): | ||
| has_telemetry_decorator = False | ||
| for decorator in node.decorator_list: | ||
| if ( | ||
| hasattr(decorator, "id") | ||
| and decorator.id == telemetry_decorator_name | ||
| ): | ||
| has_telemetry_decorator = True | ||
| break | ||
| if not has_telemetry_decorator: | ||
| failed_funcs.append(node.name) | ||
| if len(failed_funcs) > 0: | ||
| raise DecoratorError( | ||
| f"functions {failed_funcs} should be decorated with {telemetry_decorator_name}" | ||
| ) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| check_standalone_function_snowpark_pandas_telemetry_decorator( | ||
| target_file="src/snowflake/snowpark/modin/pandas/io.py", | ||
| telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator", | ||
| ) | ||
| check_standalone_function_snowpark_pandas_telemetry_decorator( | ||
| target_file="src/snowflake/snowpark/modin/pandas/general.py", | ||
| telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator", | ||
| ) | ||
| check_standalone_function_snowpark_pandas_telemetry_decorator( | ||
| target_file="src/snowflake/snowpark/modin/plugin/extensions/pd_extensions.py", | ||
| telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator", | ||
| ) | ||
| check_standalone_function_snowpark_pandas_telemetry_decorator( | ||
| target_file="src/snowflake/snowpark/modin/plugin/extensions/pd_overrides.py", | ||
| telemetry_decorator_name="snowpark_pandas_telemetry_standalone_function_decorator", | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.