[core] introduce deletedIndexFiles in IndexIncrement and allow multi … #3368
Conversation
…index files in the same partition and bucket
YannByron
force-pushed
the
master_dv_index
branch
from
5c6026e to
4b4eef7
Compare
| public Map<String, DeletionVector> readAllDeletionVectors(IndexFileMeta fileMeta) { | ||
| LinkedHashMap<String, Pair<Integer, Integer>> deletionVectorRanges = | ||
| fileMeta.deletionVectorsRanges(); | ||
| if (deletionVectorRanges == null || deletionVectorRanges.isEmpty()) { |
There was a problem hiding this comment.
Should this file be deleted? Why should handle here?
There was a problem hiding this comment.
yep, this IndexFileMeta must be Deletion_Vector Index, and ranges is not empty. Can remove this part.
| } | ||
| } | ||
|
|
||
| /** We combine the previous and new index files by {@link IndexManifestEntry#identifier}. */ |
There was a problem hiding this comment.
Just move IndexManifestEntry#identifier to here? Actually it is not identifier now. It is only for non-UnawareBucket table.
There was a problem hiding this comment.
this method can remove here.
| List<IndexManifestEntry> prevIndexFiles, List<IndexManifestEntry> newIndexFiles) { | ||
| Map<IndexManifestEntry.Identifier, IndexManifestEntry> indexEntries = | ||
| prevIndexFiles.stream() | ||
| .filter(entry -> entry.kind() == FileKind.ADD) |
There was a problem hiding this comment.
prevIndexFiles should has not delete files. here just checkArguement?
(maybe you can assert outside)
| List<IndexManifestEntry> prevIndexFiles, List<IndexManifestEntry> newIndexFiles) { | ||
| Map<String, IndexManifestEntry> indexEntries = new LinkedHashMap<>(); | ||
| for (IndexManifestEntry entry : prevIndexFiles) { | ||
| if (entry.kind() == FileKind.ADD) { |
There was a problem hiding this comment.
prevIndexFiles should has not delete files. here just checkArguement?
(maybe you can assert outside)
There was a problem hiding this comment.
yes. all the prevIndexFiles should be FileKind.ADD.
| Map<String, IndexManifestEntry> indexEntries = new LinkedHashMap<>(); | ||
| for (IndexManifestEntry entry : prevIndexFiles) { | ||
| if (entry.kind() == FileKind.ADD) { | ||
| indexEntries.put(entry.indexFile().fileName(), entry); |
There was a problem hiding this comment.
Why not using deletionVectorsRanges?
I think fileName is wrong key.
There was a problem hiding this comment.
It can be used in both Hash and Deletion_Vector index files.
|
+1 |
…index files in same bucket (apache#3368)
…index files in same bucket (apache#3368)
…index files in the same partition and bucket
Purpose
Linked issue: close #xxx
Tests
API and Format
Documentation