[spark] Add EvalSubqueriesForDeleteTable for quick delete with subqueries #3464
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
YannByron
reviewed
Jun 3, 2024
...src/main/scala/org/apache/paimon/spark/catalyst/optimizer/EvalSubqueriesForDeleteTable.scala
Show resolved
Hide resolved
YannByron
reviewed
Jun 3, 2024
| override def apply(plan: LogicalPlan): LogicalPlan = { | ||
| plan.transformDown { | ||
| case d @ DeleteFromPaimonTableCommand(_, table, condition) | ||
| if SubqueryExpression.hasSubquery(condition) && |
There was a problem hiding this comment.
Should we consider the case that this condition contains the partition predicate with subquery and other data predicates. I think taking this case into account may reduce the splits we have to scan.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Add new rule
EvalSubqueriesForDeleteTable:For those delete conditions with subqueries that only contain partition columns, we can eval them
in advance. So that when running
DeleteFromPaimonTableCommand, we can directly calldropPartitions to achieve fast deletion.
Tests
Test on 100T tpcds's DF_SS q2: before 100s+, after 2s
API and Format
Documentation