SNOW-1417060: Pylance throws errors when using copy_options with copy_into_location method

[Yoshi-Egawa]

Please answer these questions before submitting your issue. Thanks!

1. What version of Python are you using?

   Python 3.11.8

2. What operating system and processor architecture are you using?

   linux 64

3. What are the component versions in the environment (pip freeze)?

   snowflake-snowpark-python==1.16.0

4. What did you do?

   Let's say you have a custom class like the one at the bottom (a minimal example to check for errors).

from snowflake.snowpark.session import Session

connection_parameters: dict[str, int | str] = {
    "user": "<user_name>",
    "password": "<password>",
    "account": "<account_name>",
    "role": "<role_name>",
    "warehouse": "<warehouse_name>",
    "database": "<database_name>",
    "schema": "<schema_name>",
}

session = Session.builder.configs(connection_parameters).create()

df = session.create_dataframe(
    [["John", "Berry"], ["Rick", "Berry"], ["Anthony", "Davis"]],
    schema=["FIRST_NAME", "LAST_NAME"],
)

remote_file_path = f"{session.get_session_stage()}/names.parquet"

copy_result = df.write.copy_into_location(
    remote_file_path,
    file_format_type="parquet",
    header=True,
    overwrite=True,
    single=True,
)

When linting this file (Pylance v2024.4.1) errors appear.

No overloads for "copy_into_location" match the provided arguments

Argument of type "Literal[True]" cannot be assigned to parameter "overwrite" of type "str | None" in function "copy_into_location"
  Type "Literal[True]" cannot be assigned to type "str | None"
    "Literal[True]" is incompatible with "str"
    "Literal[True]" is incompatible with "None"

Argument of type "Literal[True]" cannot be assigned to parameter "single" of type "str | None" in function "copy_into_location"
  Type "Literal[True]" cannot be assigned to type "str | None"
    "Literal[True]" is incompatible with "str"
    "Literal[True]" is incompatible with "None"

6. What did you expect to see?

   The type of copy_options in the copy_into_location method is Optional[str], but the type of copy_options in the CopyIntoLocationNode class used in the copy_into_location method is Dict[str, Any]. Consequently, to align the types, the copy_options in the copy_into_location method should be of type Optional[Any].

7. Can you set logging to DEBUG and collect the logs?

   This is not a RunTime error, so I think this will not give us any extra information.

[Yoshi-Egawa]

I made a mistake with this PR(#1372).

The copy_options of the copy_into_location method was supposed to be of type Optional[Any], but I changed it to Optional[Dict[str, Any]].

[sfc-gh-sghosh]

Hello @Yoshi-Egawa ,

The PR #1372 already merged, so you would like to undo the changes and propose a new fix ( The copy_options of the copy_into_location method was supposed to be of type Optional[Any], but I changed it to Optional[Dict[str, Any]].) ?

Regards,
Sujan

[Yoshi-Egawa]

Hello @sfc-gh-sghosh ,

You're absolutely right. My apologies for the error in PR #1372. I've reverted the incorrect changes and opened a new pull request (PR #1575) with a more robust type definition for copy_options. This PR replaces the previous copy_options type with a stricter TypedDict definition, ensuring more accurate and robust type checking.

Thanks for catching that!
Yoshi

[sfc-gh-sghosh]

Hello @Yoshi-Egawa,

Thanks for submitting the PR (PR #1575).

Regards,
Sujan

[IraSil9]

@sfc-gh-sghosh @Yoshi-Egawa

Hello everyone.
I think there might still be an issue with this.

I'm using:
snowflake-snowpark-python = 1.20.0
(this is the latest version at the time of writing this comment)

The copy_into_location function expects **copy_options: Optional[Dict[str, Any]].
I provide: copy_options={"MAX_FILE_SIZE": 134217728}

On Snowflake side, the COPY statement itself gets translated like this:

COPY  INTO...
FROM ... 
FILE_FORMAT  = (  TYPE  = parquet )  
copy_options = {'MAX_FILE_SIZE': '134217728'}  
HEADER  = True

This produces an error since this is not the expected way to provide copy options.

In PR 1575 the type was supposed to be changed but this PR never got merged. It's closed.

[Yoshi-Egawa]

Hello @sfc-gh-sghosh ,

Apologies, but it seems that I accidentally closed PR #1575. I’ve created a new PR (#1828) for the same changes. Could you kindly take a look at the new PR when you have a moment?

Thank you for your understanding and support!

Best regards,
Yoshi Egawa