[spark] Support report scan ordering #4026
Conversation
|
Thanks @ulysses-you for the contribution. This optimization looks good to me! But there is some potential issues. |
| /** | ||
| * To indicate if this `DataSplit` keep the raw table ordering. For example, for the primary key | ||
| * table, we will do sorted run during write and do merge read during read, so the data is | ||
| * sorted by the primary keys. Return `false` means the ordering is broken. If a `DataSplit` is | ||
| * `rawConvertible` then there is no sort merge read, so only if the data file number less than | ||
| * 2, we can get the correct data ordering. | ||
| */ | ||
| public boolean keepOrdering() { | ||
| return !rawConvertible || dataFiles.size() < 2; | ||
| } |
There was a problem hiding this comment.
cc @JingsongLi , If I get it correctly, this should address your concern ?
| return Array.empty | ||
| } | ||
|
|
||
| val allSplitsKeepOrdering = lazyInputPartitions.toSeq |
There was a problem hiding this comment.
Maybe we should make sure there is only one split in a bucket?
There was a problem hiding this comment.
thank you @JingsongLi, please correct me if wrong. Per my understanding, if a partition contains multi-splits using merge file read, then all splits should be ordered and never be overlapped ?
There was a problem hiding this comment.
Merging only occurs within a single split.
|
|
||
| import scala.collection.JavaConverters._ | ||
|
|
||
| case class PaimonScan( |
There was a problem hiding this comment.
Is there possibility to extract a base class PaimonBucketedScan to reuse codes?
| * 2, we can get the correct data ordering. | ||
| */ | ||
| public boolean keepOrdering() { | ||
| return !rawConvertible || dataFiles.size() < 2; |
There was a problem hiding this comment.
Maybe we don't need to introduce method here, just use !rawConvertible || dataFiles.size() < 2 in spark.
|
thank you @JingsongLi , addressed comments |
Purpose
This pr makes
PaimonScansupportSupportsReportOrdering. For primary key table, we will do a sorted run during write so the primary keys in one file are always ordered. And skip report ordering if it contains a rawConvertible split since we will not do merge sort read.This can help eliminate local sort for smj/sorted agg etc..
Tests
add tests
API and Format
no
Documentation