Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BLAZE-707][FOLLOWUP] NativePaimonTableScanExec should use shimed PartitionedFile and min partition number #713

Merged
merged 1 commit into from
Dec 21, 2024

Conversation

SteNicholas
Copy link
Contributor

Which issue does this PR close?

Closes #707.

Rationale for this change

PartitionedFile has different constructors among different Spark versions. Meanwhile, the min partition number of max splits is also different for different Spark versions. NativePaimonTableScanExec should use shimed PartitionedFile and min partition number for compilation error as follows:

  • spark-3.0
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:232: value filesMinPartitionNum is not a member of org.apache.spark.sql.internal.SQLConf

  • spark-3.4, spark-3.5
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:219: type mismatch;
 found   : String
 required: org.apache.spark.paths.SparkPath
Error: ] /home/runner/work/blaze/blaze/spark-extension/src/main/scala/org/apache/spark/sql/hive/execution/blaze/plan/NativePaimonTableScanExec.scala:222: type mismatch;
 found   : String
 required: org.apache.spark.paths.SparkPath

What changes are included in this PR?

NativePaimonTableScanExec uses shimed PartitionedFile and min partition number to fix compilation error.

Are there any user-facing changes?

No.

@richox
Copy link
Collaborator

richox commented Dec 21, 2024

seems ci is still broken, i will fix it later.

@richox richox merged commit 43e3621 into kwai:master Dec 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support native scan paimon cow table
2 participants