Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support spark 3.5 #1142

Open
melin opened this issue Dec 11, 2023 · 2 comments
Open

Support spark 3.5 #1142

melin opened this issue Dec 11, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@melin
Copy link

melin commented Dec 11, 2023

Support spark 3.5
@davidrabinowitz
Copy link
Member

The spark-3.4-bigquery connector can be used on Spark 3.5. Are there any specific spark 3.5 features you need to be supported?

@melin
Copy link
Author

melin commented Dec 11, 2023

The spark-3.4-bigquery connector can be used on Spark 3.5. Are there any specific spark 3.5 features you need to be supported?

We're excited about some of the improvements in Spark 3.5, e.g.:

Allow ShuffleDriverComponent to declare if shuffle data is reliably stored SPARK-42689
Allow V2 writes to indicate advisory shuffle partition size SPARK-42779
Any plan on support Spark 3.5?

@isha97 isha97 added the enhancement New feature or request label Feb 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@davidrabinowitz @melin @isha97