Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No module named 'delta_kernel_rust_sharing_wrapper' on delta-sharing 1.1.0 #552

Open
championj-db opened this issue Aug 5, 2024 · 6 comments

Comments

@championj-db
Copy link

championj-db commented Aug 5, 2024

Issue
After pip install delta-sharing I cannot run an import delta_sharing command as I receive this error:
ModuleNotFoundError: No module named 'delta_kernel_rust_sharing_wrapper'

Reverting back to 1.0.5 with pip install delta-sharing==1.0.5 appears to resolve this issue, As does doing a pip install delta-kernel-rust-sharing-wrapper

Expectation
Doing a pip install delta-sharing on the latest version should be able to pull in required dependencies

Context
I am trying to run the delta-sharing-airlines demo https://www.databricks.com/resources/demos/tutorials/data-sharing/delta-sharing-airlines and encounter this in notebook 03-receiver-delta-sharing-demo using the generated cluster (DBR 12.2 LTS ML, Spark 3.3.2, Scala 2.12)

@forzagreen
Copy link

As explained in the release notes of v1.1.0, section Important Installation Notes, you must install delta-kernel-rust-sharing-wrapper

I agree that it would be better if delta-kernel-rust-sharing-wrapper were a dependancy of delta-sharing.

@GBBBAS
Copy link

GBBBAS commented Aug 5, 2024

As explained in the release notes of v1.1.0, section Important Installation Notes, you must install delta-kernel-rust-sharing-wrapper

I agree that it would be better if delta-kernel-rust-sharing-wrapper were a dependancy of delta-sharing.

Can we expect that it is it planned to be added as a dependency in a next release?

@GeroSalas
Copy link

still experiencing issue with this

image

here is my Dockerfile

FROM public.ecr.aws/lambda/python:3.9

# Verify that python3 is installed correctly
RUN python3 --version
RUN python3 -m pip install --no-cache-dir --upgrade pip
RUN python3 -m pip install pyodbc

# Update the package list and install glibc 2.31+ or higher
RUN yum -y update && yum -y install glibc
# Verify glibc version
RUN ldd --version

#WORKDIR ${LAMBDA_TASK_ROOT}

COPY index.py ${LAMBDA_TASK_ROOT}
COPY requirements.txt ${LAMBDA_TASK_ROOT}
COPY src ${LAMBDA_TASK_ROOT}/src

# Delta-Sharing Release Notes
# https://github.com/delta-io/delta-sharing?tab=readme-ov-file#installation
# https://github.com/delta-io/delta-sharing/releases/tag/py-v1.1.0
RUN python3 -m pip install delta-kernel-rust-sharing-wrapper

RUN python3 -m pip install --upgrade -r requirements.txt -t ${LAMBDA_TASK_ROOT}

CMD [ "index.handler" ]

@cvargas-xbrein
Copy link

same error from aws lambda python 3.10 when try to installing deps on efs storage

 ERROR: Could not find a version that satisfies the requirement delta-kernel-rust-sharing-wrapper (from delta-sharing) (from versions: none)
ERROR: No matching distribution found for delta-kernel-rust-sharing-wrapper

@xy1024xiangyu
Copy link

xy1024xiangyu commented Nov 26, 2024

same error here. Also tried to install from the whl file manually, but seems currently not supporting AWS EC2 x86_64 instance?

also tried to follow this https://github.com/delta-io/delta-sharing/blob/main/python/delta-kernel-rust-sharing-wrapper/README.md
cannot figure out a solution...

@mattaop
Copy link

mattaop commented Nov 26, 2024

I had the same issue when running spark 3.3, but when running spark 3.4 the issue disappeared. Then delta-kernel-rust-sharing-wrapper was installed as part of delta-sharing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants