Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[spark] Fix can not set bucket number in HiveMigrator #3332

Closed
wants to merge 1 commit into from

Conversation

xuzifu666
Copy link
Member

Fix can not set bucket number in HiveMigrator,currently in MigrateTableProcedure can only set bucket = -1, user cannot set bucekts,the pr is aim to fix it.

Linked issue: close #xxx

Tests

API and Format

Documentation

@JingsongLi
Copy link
Contributor

Hi @xuzifu666 , why do you want to set bucket number for this case?

@xuzifu666
Copy link
Member Author

Hi @xuzifu666 , why do you want to set bucket number for this case?
Thanks for your attention. Spark load to paimon table is more fast with setting buckets, user can set bucket number according to the total data size.So in my option give user a way to set bucket number is a better way.If my view is wrong please correct me anytime😃@JingsongLi

@JingsongLi
Copy link
Contributor

@xuzifu666

Spark load to paimon table is more fast with setting buckets

What is this come from? Do you have some reproduce steps?

@xuzifu666
Copy link
Member Author

@xuzifu666

Spark load to paimon table is more fast with setting buckets

What is this come from? Do you have some reproduce steps?
This is in our test spark data to paimon,if bucket is -1,would had a bucket assgine, so user would set bucket number avoid bucket assgine, mainly in the scene.@JingsongLi

@xuzifu666
Copy link
Member Author

Keep the default is better,so close the pr first

@xuzifu666 xuzifu666 closed this May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants