-
Notifications
You must be signed in to change notification settings - Fork 116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-869536 Fix buggy behavior in DataFrame.to_local_iterator #1226
Conversation
c3d8632
to
3d52ae8
Compare
3d52ae8
to
f3962af
Compare
) -> Dict[str, Any]: | ||
if to_iter and not to_pandas: # Fix for SNOW-869536 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to_pandas
doesn't have this issue, SnowflakeCursor.fetch_pandas_batches
already handles the isolation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we put this into comment instead? This seems more informative than "fix for" :)
@@ -434,8 +434,14 @@ def _to_data_or_iter( | |||
results_cursor: SnowflakeCursor, | |||
to_pandas: bool = False, | |||
to_iter: bool = False, | |||
num_statements: Optional[int] = None, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Did we remove this because the param is unused?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes
) -> Dict[str, Any]: | ||
if to_iter and not to_pandas: # Fix for SNOW-869536 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we put this into comment instead? This seems more informative than "fix for" :)
1f752cc
to
a524268
Compare
Please answer these questions before submitting your pull requests. Thanks!
What GitHub issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-869536: Iterators from to_local_iterator stop returning results after another query occurs #945
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
Issue: All queries issued from a Snowpark
Session
object are executed using the same SnowflakeCursor instance, this causes unexpected, wrong behavior when we fetch results from an iterator but have code that executes a different query half way, e.g.Fix: This PR changes
src/snowflake/snowpark/_internal/server_connection.py
create a new cursor object locally to read data from the last executed query and continues to use this cursor object in the iterator to achieve isolation between the iterator and the following queries executed within the session.