[core] Filter data file while executing local orphan clean #4287
Conversation
| candidateDeletes.removeAll(usedFiles); | ||
| candidateDeletes.stream().map(candidates::get).forEach(fileCleaner); | ||
| deleteFiles.addAll( | ||
| candidateDeletes.stream().map(candidates::get).collect(Collectors.toList())); | ||
|
|
There was a problem hiding this comment.
candidateDeletes.clear();
|
I think this is a good optimization, but two questions. |
|
I'm not sure whether this optimization can be used in a distributed situation, because that means that the candidate set needs to be broadcast to each executor, bring additional consumption. At this time, the number of data file meta in each executor is not large, and there is no risk of OOM, unless there is a particularly serious case of data skew On the other hand, the process of joining condidate set and used file set actually implicitly includes the process of broadcasting and filtering. Therefore, the optimization benefits needs to be further analyzed. And the analyze in production environment with LocalOrphanClean will be given later. |
|
Test in table with 7TB size with local mode: Before this patch, it will OOM when it collect 80 million data file metas: After optimization, it only collect about 2000 manifests and about 60000 data file metas, the procedure only cost 20min on local machine: |
| @@ -114,7 +118,7 @@ private List<String> getUsedFiles(String branch) { | |||
| ManifestFile manifestFile = | |||
| table.switchToBranch(branch).store().manifestFileFactory().create(); | |||
| try { | |||
| List<String> manifests = new ArrayList<>(); | |||
| Set<String> manifests = new HashSet<>(); | |||
There was a problem hiding this comment.
While reading multiple snapshots, there will be duplicate manifests. Deduplication can improve efficiency.
|
Could you review it again? thanks @wwj6591812 |
Purpose
Local Remove Orphan Files OOM always occurred when collecting data file meta. Because data files in tables may be large (could reach tens of millions or more).
Relatively speaking, the number of candidate files is not large (ten thousand or hundreds of thousands).
So we can just collect data file meta in candidate files to reduce the probability of OOM occurrence.
Tests
API and Format
Documentation