[core] Manage the lookup file disk cache in task granularity #3853
Conversation
| @@ -207,12 +211,8 @@ private LookupFile createLookupFile(DataFileMeta file) throws IOException { | |||
| return new LookupFile(localFile, file, lookupStoreFactory.createReader(localFile, context)); | |||
| } | |||
|
|
|||
| @Override | |||
| public void close() throws IOException { | |||
There was a problem hiding this comment.
Should we clean all files when closing?
There was a problem hiding this comment.
In here . And if a partition bucket writer is idle and closed, the lookup file for this writer will only be evicted based on size or LRU (there is no explicit close for this situation).
There was a problem hiding this comment.
But by default, there is no max size configuration.
There was a problem hiding this comment.
Evict by the retention time?
There was a problem hiding this comment.
Before the LookupLevels's cache file will be cleared when the writer is idled (no data written to in the latest snapshot).
Now, the cache file will still be there after the idle writer closed and evict by the retention time by default.
There was a problem hiding this comment.
Yes, it is better to remove all files when closing this writer.
There was a problem hiding this comment.
You can manage a collection in LookupLevels.
There was a problem hiding this comment.
You can manage a collection in
LookupLevels.
Thanks, Make sense to me.
Purpose
Linked issue: close #3848
Tests
API and Format
Documentation