SNOW-1269037: NaT (Not a Time) not parsed as NULL when session.create_dataframe (throws exception instead)

[cvmartin]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?

   3.11.7

2. What are the Snowpark Python and pandas versions in the environment?

   pandas==1.5.3
   snowflake-snowpark-python==1.12.0

3. What did you do?

The following code works as expected:

from snowflake.snowpark.types import StructType, StructField, TimestampType

from snowflake.snowpark import Session
import pandas as pd

session = Session.builder.config('local_testing', True).create()

df = pd.DataFrame(
    {
        "date": pd.to_datetime(
            ["2020-01-02", "2020-01-13", "2020-02-01", "2020-02-23", "2020-03-05"],
            utc=True
        )
    }
)

sf_df = session.create_dataframe(
    data = df,
    schema = StructType([StructField("date", TimestampType())])
)
sf_df.schema

I get StructType([StructField('"date"', TimestampType(), nullable=True)]). Specifying the schema is not even necessary. What is important is specifying utc=True; otherwise the column is coerced to a LongType().

Now, if one of the values of the date vector is coerced into a NaT value (not a time):

df = pd.DataFrame(
    {
        "date": pd.to_datetime(
            [None, "2020-01-13", "2020-02-01", "2020-02-23", "2020-03-05"],
            utc=True
        )
    }
)

sf_df = session.create_dataframe(
    data = df,
    schema = StructType([StructField("date", TimestampType())])
)

The snippet above throws a very clear exception, indicating that NaTs are not supported:

TypeError: not supported type: <class 'pandas._libs.tslibs.nattype.NaTType'>

4. What did you expect to see?

This behavior does not happen out of the Local Testing framework. Using a "real snowflake session", the dataframe is properly created, parsing NaT as NULL:

-----------------------------
|"date"                     |
-----------------------------
|NULL                       |
|2020-01-12 16:00:00-08:00  |
|2020-01-31 16:00:00-08:00  |
|2020-02-22 16:00:00-08:00  |
|2020-03-04 16:00:00-08:00  |
-----------------------------

Also, NaN values work alright inside the Local Testing framework. In other words, if the snipped above would have used a numeric column, it would have worked fine. It is only with time values (NaT) when the problem arises.

[sfc-gh-sghosh]

Hello @cvmartin ,

Thanks for raising the issue, we are looking into it.

Regards,
Sujan

[sfc-gh-sghosh]

Hello @cvmartin ,

I tried the above sample code snippet with snowpatk 1.14.0 and its not throwing any error, following is the output

code:
`
from snowflake.snowpark.types import StructType, StructField, TimestampType
from snowflake.snowpark import Session
import pandas as pd

df = pd.DataFrame(
{
"date": pd.to_datetime(
[None, "2020-01-13", "2020-02-01", "2020-02-23", "2020-03-05"],
utc=True
)
}
)

sf_df = session.create_dataframe(
data=df,
schema=StructType([StructField("date", TimestampType())])
)

sf_df.schema
`

Output:
StructType([StructField('"date"', TimestampType(tz=ltz), nullable=True)])

For both case whether I use None or valid date value, the output is same as above

Regards,
Sujan

[cvmartin]

Hello @sfc-gh-sghosh.

Thanks for your reply. The code that you provide I believe it does not make use of the Local Testing Framework, which is where I find the problem.

Specifically, I miss the line

session = Session.builder.config('local_testing', True).create()

That indicates to create a Local Testing Framework connection named session (else, NameError: name 'session' is not defined)

With the complete code, that looks like

from snowflake.snowpark.types import StructType, StructField, TimestampType
from snowflake.snowpark import Session
import pandas as pd

session = Session.builder.config('local_testing', True).create()


df = pd.DataFrame(
{
"date": pd.to_datetime(
[None, "2020-01-13", "2020-02-01", "2020-02-23", "2020-03-05"],
utc=True
)
}
)

sf_df = session.create_dataframe(
data=df,
schema=StructType([StructField("date", TimestampType())])
)

sf_df.schema

I can reproduce the error, also with snowflake-snowpark-python==1.14.0

[sfc-gh-sghosh]

Thank you @cvmartin ,

We are able to reproduce the issue, we are looking into it, will update.

Regards,
Sujan

[sfc-gh-sghosh]

Hello @cvmartin ,

The team is working on the fix via #1393

Regards,
Sujan

[cvmartin]

Thanks a lot!

[sfc-gh-dszmolka]

PR is merged and will be part of the next release

[sfc-gh-aling]

I'm closing the issue as the PR has been merged the released. please feel free to reach out if you still see issues.