added greater and less than query optimization to snapshot table queries #3396
Conversation
| if (snapshotPred != null && snapshotPred.function() instanceof Equal) { | ||
| specificSnapshot = (Long) snapshotPred.literals().get(0); | ||
| if (snapshotPred != null) { | ||
| if (snapshotPred.function() instanceof Equal) { |
There was a problem hiding this comment.
This will cause performance degradation of equal, because this requires list files
There was a problem hiding this comment.
Originally I had preserved the efficiency in snapshotsWithinRange, with the new code removing the list call it should still be efficient with the iterating on a single snapshopId
There was a problem hiding this comment.
Thanks @ctrlaltdilj for the contribution!
Left one comment.
| return Collections.singletonList(snapshot(maxSnapshotId)).iterator(); | ||
| } | ||
|
|
||
| return listVersionedFiles(fileIO, snapshotDirectory(), SNAPSHOT_PREFIX) |
There was a problem hiding this comment.
We still need to list directory.
Can we directly read one by one?
We can combine maxSnapshotId, minSnapshotId, earliest and latest. Calculate a queryable interval and directly call the snapshot method one by one.
There was a problem hiding this comment.
Removed list call, lmk what you think
optimizing queries on snapshot table