snowflake-connector-python fails in datahub-ingestion base image due to oscrypto bug

[lzrlee]

Describe the bug
After upgrading to 0.12.0 and using a snowflake ingestion yaml, we are receiving the following error when connecting 'snowflake-connection': ['Failed to connect to snowflake instance due to error Error detecting the version of libcrypto.']

This is due to an issue with the oscrypto dependency in snowflake-python-connect:

• Openssl version not detected in current oscrypto version (pypi) wbond/oscrypto#78
• https://community.snowflake.com/s/article/Python-Connector-fails-to-connect-with-LibraryNotFoundError-Error-detecting-the-version-of-libcrypto#openssl

To Reproduce
Steps to reproduce the behavior:

1. Deploy a datahub-ingest-cron cronjob for Snowflake
2. Run a manual job: `kubectl create job --from=cronjob/
3. read logs of the cron job and see error logs

Expected behavior
Snowflake Ingestion job to succeed correctly

Screenshots
Logs:

[2023-11-06 19:52:52,783] INFO     {datahub.cli.ingest_cli:147} - DataHub CLI version: 0.12.0+docker
[2023-11-06 19:52:53,069] INFO     {datahub.ingestion.run.pipeline:213} - Sink configured successfully. DataHubRestEmitter: ****
[2023-11-06 19:53:06,058] INFO     {datahub.ingestion.source_config.sql.snowflake:139} - using authenticator type 'DEFAULT_AUTHENTICATOR'
[2023-11-06 19:53:06,299] INFO     {datahub.ingestion.run.pipeline:230} - Source configured successfully.
[2023-11-06 19:53:06,300] INFO     {datahub.cli.ingest_cli:128} - Starting metadata ingestion
[2023-11-06 19:53:06,673] ERROR    {datahub.ingestion.source.snowflake.snowflake_v2:234} - snowflake-connection => Failed to connect to snowflake instance due to error Error detecting the version of libcrypto.
[2023-11-06 19:53:06,956] INFO     {datahub.ingestion.source.state.checkpoint:145} - Successfully constructed last checkpoint state for job common_ingest_from_sql_source with timestamp 2023-10-05 12:28:10.564000+00:00
[2023-11-06 19:53:06,959] INFO     {datahub.ingestion.run.pipeline:476} - Processing commit request for DatahubIngestionCheckpointingProvider. Commit policy = CommitPolicy.ON_NO_ERRORS, has_errors=True, has_warnings=False
[2023-11-06 19:53:06,959] WARNING  {datahub.ingestion.run.pipeline:485} - Skipping commit request for DatahubIngestionCheckpointingProvider since policy requirements are not met.
[2023-11-06 19:53:06,988] INFO     {datahub.cli.ingest_cli:141} - Finished metadata ingestion

Cli report:
{'cli_version': '0.12.0+docker',
 'cli_entry_location': '/datahub-ingestion/.local/lib/python3.10/site-packages/datahub/__init__.py',
 'py_version': '3.10.13 (main, Oct 12 2023, 10:20:20) [GCC 12.2.0]',
 'py_exec_path': '/usr/local/bin/python',
 'os_details': 'Linux-5.15.134-x86_64-with-glibc2.36',
...

Source (snowflake) report:
{

...


 'failures': {'snowflake-connection': ['Failed to connect to snowflake instance due to error Error detecting the version of libcrypto.'],
              'stale-entity-removal': ['Skipping stale entity soft-deletion because the source produced no events. This is a fail-safe mechanism to prevent accidental deletion of all entities.']},
....

Sink (datahub-rest) report: ...

 Pipeline finished with at least 2 failures; produced 0 events in 0.83 seconds.

Desktop (please complete the following information):

• OS: linux (k8s Docker)
• Browser N/a
• Version Datahub 0.12.0

Additional context

• This issue is fixed in snowflake-connector-python v3.4.0: https://github.com/snowflakedb/snowflake-connector-python/blob/main/DESCRIPTION.md#release-notes

[github-actions]

This issue is stale because it has been open for 30 days with no activity. If you believe this is still an issue on the latest DataHub release please leave a comment with the version that you tested it with. If this is a question/discussion please head to https://slack.datahubproject.io. For feature requests please use https://feature-requests.datahubproject.io

[rospe]

still not fixed in v0.12.1. We cannot deploy to prod until this is resolved.

[tkdrahn]

Getting same issue with v0.11.0.5 . It works fine with v0.11.0.4

[hsheth2]

It looks like this was fixed in snowflake-connector-python v3.4.0. We'll be updating our base images shortly.