Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-1655751: ERROR with GenericFunction 'flatten' is already registered and is going to be overridden #530

Closed
nadavuoptimove opened this issue Sep 8, 2024 · 12 comments
Assignees
Labels
bug Something isn't working status-fixed_awaiting_release The issue has been fixed, its PR merged, and now awaiting the next release cycle of the connector. status-triage_done Initial triage done, will be further handled by the driver team

Comments

@nadavuoptimove
Copy link

Hi.

  1. What version of Python are you using?

Python 3.9.6 (default, Feb 3 2024, 15:58:27)

  1. What operating system and processor architecture are you using?

macOS-14.5-arm64-arm-64bit (it happens also when running it on docker --platform=linux/amd64 public.ecr.aws/docker/library/python:3.11.7-slim)

  1. What are the component versions in the environment (pip freeze)?

asn1crypto==1.5.1
cachetools==5.5.0
certifi==2024.8.30
cffi==1.17.1
charset-normalizer==2.1.1
cryptography==36.0.2
exceptiongroup==1.2.2
filelock==3.16.0
google-api-core==2.19.2
google-auth==2.34.0
google-cloud-core==2.4.1
google-cloud-pubsub==2.23.0
google-cloud-secret-manager==2.20.2
google-cloud-storage==2.18.2
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.65.0
grpc-google-iam-v1==0.13.1
grpcio==1.66.1
grpcio-status==1.66.1
idna==3.8
iniconfig==2.0.0
numpy==1.26.4
packaging==24.1
pandas==1.5.3
platformdirs==4.3.1
pluggy==1.5.0
proto-plus==1.24.0
protobuf==5.28.0
psycopg2==2.9.9
pyasn1==0.6.0
pyasn1_modules==0.4.0
pycparser==2.22
PyJWT==2.9.0
pyodbc==4.0.39
pyOpenSSL==22.0.0
pysnowflake==0.1.3
pytest==7.4.0
pytest-mock==3.10.0
python-dateutil==2.9.0.post0
python-dotenv==1.0.0
python-json-logger==2.0.7
pytz==2024.1
requests==2.28.1
rsa==4.9
six==1.16.0
snowflake-connector-python==3.12.1
snowflake-sqlalchemy==1.6.1
sortedcontainers==2.4.0
SQLAlchemy==2.0.34
tomli==2.0.1
tomlkit==0.13.2
tornado==6.4.1
typing_extensions==4.12.2
urllib3==1.26.20

  1. What did you do?

we have number of different python micro services that run on K8, lately we are seeing an issue when running this jobs:

SAWarning: The GenericFunction 'flatten' is already registered and is going to be overridden.
  functions.register_function("flatten", flatten)

This jobs are using the most updated version of snowflake-sqlalchemy and sqlalchemy.

We think the issue is coming from the following line that was pushed in the following commit :

d78f0c0

image

Before this commit we didn't have this error.

  1. Steps to reproduce:
    Install the pypi packages we provided and start running python code, the error will show at the beginning.

Waiting to hear from you.

Thanks,
Nadav

@nadavuoptimove nadavuoptimove added bug Something isn't working needs triage labels Sep 8, 2024
@github-actions github-actions bot changed the title ERROR with GenericFunction 'flatten' is already registered and is going to be overridden SNOW-1655751: ERROR with GenericFunction 'flatten' is already registered and is going to be overridden Sep 8, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka self-assigned this Sep 9, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added status-triage Issue is under initial triage and removed needs triage labels Sep 9, 2024
@sfc-gh-dszmolka
Copy link
Contributor

hi and thank you for raising this issue with us - does the above sentence

Install the pypi packages we provided and start running python code, the error will show at the beginning.

mean that you wanted to attach something (repro script, etc) to this issue? If so, it would be immensely helpful to have it. Thank you in advance !

@sfc-gh-dszmolka sfc-gh-dszmolka added the status-information_needed Additional information is required from the reporter label Sep 9, 2024
@nadavuoptimove
Copy link
Author

nadavuoptimove commented Sep 9, 2024

Hi,

No, because this error occurs in many different python jobs, so the python code you are running is not important.

Thanks

@top-jvn
Copy link

top-jvn commented Sep 10, 2024

Hi,

We experience the same problem. Example:

from typing import Any

from snowflake.connector import connect
from sqlalchemy import create_engine, text


def connection_factory():
    connection_params = {
        "account": ...,
        "user": ...,
        "role": ...,
        "warehouse": ...,
        "database": ...,
        "schema": ...,
        "password": ...,
    }
    return connect(**connection_params)


def run():
    engine: Any = create_engine("snowflake://", creator=connection_factory)

    with engine.connect() as connection:
        try:
            [results] = connection.execute(
                text("select 'hello world' as msg")
            ).fetchone()
            print(results)
        finally:
            connection.close()

    engine.dispose()


if __name__ == "__main__":
    run()

@sfc-gh-dszmolka
Copy link
Contributor

thank you for the example ! we're going to take a look.

@sfc-gh-dszmolka sfc-gh-dszmolka removed their assignment Sep 10, 2024
@sfc-gh-dszmolka sfc-gh-dszmolka added status-triage_done Initial triage done, will be further handled by the driver team status-in_progress Issue is worked on by the driver team and removed status-information_needed Additional information is required from the reporter status-triage Issue is under initial triage labels Sep 10, 2024
@sfc-gh-dszmolka
Copy link
Contributor

sfc-gh-dszmolka commented Sep 16, 2024

work in progress - draft PR open at #532 . Please bear with us while its fully worked out and merged (and a fixed connector is released) I'll keep this thread posted.

@catzc
Copy link

catzc commented Oct 8, 2024

Hi! I see #532 is released. Do we know when a new package will be released?

@sfc-gh-dszmolka sfc-gh-dszmolka added status-fixed_awaiting_release The issue has been fixed, its PR merged, and now awaiting the next release cycle of the connector. and removed status-in_progress Issue is worked on by the driver team labels Oct 9, 2024
@sfc-gh-dszmolka
Copy link
Contributor

Indeed the fix is merged ! At this moment I do not have any indication of the new release, but I would assume it doesn't happen before the end of this month. Thank you for bearing with us !

@nadavuoptimove
Copy link
Author

Hi !

Any updates ? we are waiting to this fix a lot of time.

Thanks!

@benvdh-incentro
Copy link

@sfc-gh-dszmolka the end of October 2024 has now been reached (actually we're 2 weeks further than that). When do you expect this fix to be released?

This warning is causing quite some stdout pollution (and possibly log polution) for many users.

@sfc-gh-dszmolka
Copy link
Contributor

sfc-gh-dszmolka commented Nov 15, 2024

You're right @benvdh-incentro , October is already way over, and I'm still unable to give you an exact release date unfortunately. I'll keep this thread posted with any new information as it becomes available to me.

edit: fresh info : release is planned after #541 is merged. So please bear with us for a little while, and you can track that PR to get an indication when this release might be happening. It looks like I cannot give a timeline estimation besides that.

edit-edit: even more info. the last release was skipped due to internal organizational changes, but per current plans are that at least 1 release should be done within this month (November)

@mrcolumbia
Copy link

How are we looking on this?

@sfc-gh-dszmolka
Copy link
Contributor

released with version 1.7.0 in November 2024 release cycle

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working status-fixed_awaiting_release The issue has been fixed, its PR merged, and now awaiting the next release cycle of the connector. status-triage_done Initial triage done, will be further handled by the driver team
Projects
None yet
Development

No branches or pull requests

7 participants