SNOW-1026241: Null value in integer column raises exception when executing .to_pandas() in version 1.12.0

[frederiksteiner]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?
   Python 3.10.13

2. What operating system and processor architecture are you using?

   Linux-5.10.102.1-microsoft-standard-WSL2-x86_64-with-glibc2.31

3. What are the component versions in the environment (pip freeze)?

   asn1crypto==1.5.1
   astroid==3.0.1
   asttokens==2.2.1
   avro==1.11.3
   backcall==0.2.0
   bcrypt==4.0.1
   black==24.1.1
   -e git+ssh://[email protected]:7999/ppp_kab/blp-elt.git@5d318c8cafeb5541760288ab39def20c02cd3f1e#egg=blp_elt
   certifi==2023.7.22
   cffi==1.15.1
   cfgv==3.3.1
   charset-normalizer==3.2.0
   classify-imports==4.2.0
   click==8.1.7
   cloudpickle==2.0.0
   comm==0.1.3
   coverage==7.4.1
   cryptography==40.0.2
   dataclasses-json==0.5.13
   debugpy==1.6.7
   decorator==5.1.1
   dill==0.3.7
   distlib==0.3.7
   dnspython==1.16.0
   et-xmlfile==1.1.0
   executing==1.2.0
   filelock==3.12.2
   freezegun==1.4.0
   identify==2.5.26
   idna==3.4
   intervaltree==3.1.0
   ipykernel==6.29.0
   ipython==8.14.0
   isort==5.12.0
   jedi==0.18.2
   jupyter_client==8.3.0
   jupyter_core==5.3.1
   kafka-python==2.0.2
   lazy-object-proxy==1.9.0
   libcst==1.0.1
   lxml==4.9.3
   marshmallow==3.20.1
   matplotlib-inline==0.1.6
   mccabe==0.7.0
   mypy-extensions==1.0.0
   nest-asyncio==1.5.6
   nodeenv==1.8.0
   numpy==1.25.1
   openpyxl==3.1.2
   oscrypto @ git+https://github.com/wbond/oscrypto.git@1547f535001ba568b239b8797465536759c742a3
   packaging==23.1
   pandas==2.1.4
   paramiko==3.4.0
   parso==0.8.3
   pathspec==0.11.1
   pexpect==4.8.0
   pickleshare==0.7.5
   platformdirs==3.9.1
   pre-commit==3.6.0
   pre-commit-hooks==4.5.0
   prompt-toolkit==3.0.39
   psutil==5.9.5
   psycopg2==2.9.9
   ptyprocess==0.7.0
   pure-eval==0.2.2
   pyarrow==10.0.1
   pycparser==2.21
   pycryptodomex==3.18.0
   pydocstyle==6.3.0
   Pygments==2.15.1
   PyJWT==2.8.0
   pylint==3.0.3
   pymongo==3.12.0
   PyNaCl==1.5.0
   pyOpenSSL==23.2.0
   pyre-check==0.9.19
   pyre-extensions==0.0.30
   python-dateutil==2.8.2
   pytz==2023.3
   PyYAML==6.0.1
   pyzmq==25.1.0
   reorder-python-imports==3.12.0
   requests==2.31.0
   ruamel.yaml==0.17.32
   ruamel.yaml.clib==0.2.7
   six==1.16.0
   snowballstemmer==2.2.0
   snowflake-connector-python==3.7.0
   snowflake-snowpark-python==1.12.0
   sortedcontainers==2.4.0
   stack-data==0.6.2
   tabulate==0.9.0
   TestSlide==2.7.1
   tomli==2.0.1
   tomlkit==0.12.0
   tornado==6.3.2
   traitlets==5.9.0
   typeguard==2.13.3
   typing-inspect==0.9.0
   typing_extensions==4.7.1
   tzdata==2023.3
   unittest-xml-reporting==3.2.0
   urllib3==1.26.16
   virtualenv==20.24.2
   wcwidth==0.2.6
   wrapt==1.15.0

4. What did you do?

   data = [
   ["String 1", 0],
   ["String 2", 1],
   ["String 3", None]
   ]
   schema = ["STRING_COL","INT_COL"]
   session.create_dataframe(data, schema).to_pandas()

5. What did you expect to see?

   No error as in the previous snowpark version.

6. Can you set logging to DEBUG and collect the logs?

2024-01-31 08:28:57,860 - MainThread cursor.py:694 - execute_helper() - DEBUG - Request id: ee19e9d3-080e-4dde-8427-e585258a7c6a
2024-01-31T07:28:57+0000 - 11042 - snowflake.connector.cursor - DEBUG - Request id: ee19e9d3-080e-4dde-8427-e585258a7c6a
2024-01-31 08:28:58,271 - MainThread connectionpool.py:474 - make_request() - DEBUG - https://.snowflakecomputing.com:443 "POST /queries/v1/query-request?requestId=ee19e9d3-080e-4dde-8427-e585258a7c6a&request_guid=84e380ba-9a03-424e-a76a-c2fc6dab8731 HTTP/1.1" 200 None
2024-01-31T07:28:58+0000 - 11042 - snowflake.connector.vendored.urllib3.connectionpool - DEBUG - https://__.snowflakecomputing.com:443 "POST /queries/v1/query-request?requestId=ee19e9d3-080e-4dde-8427-e585258a7c6a&request_guid=84e380ba-9a03-424e-a76a-c2fc6dab8731 HTTP/1.1" 200 None

The error happens in the _fix_pandas_df_fixed_type method which was recently introduced

[gudbrand3]

Experiencing same issue with error

SnowparkFetchDataException: (1406): Failed to fetch a Pandas Dataframe. The error is: Cannot convert non-finite values (NA or inf) to integer

@frederiksteiner thanks for raising!