Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Avoid building the LookupFile cache repeatedly when the cache … #3859

Merged
merged 2 commits into from
Aug 7, 2024

Conversation

Aitozi
Copy link
Contributor

@Aitozi Aitozi commented Jul 31, 2024

…is nearly full.

Purpose

Linked issue: close #3855

Tests

API and Format

Documentation

@Aitozi Aitozi requested a review from JingsongLi July 31, 2024 07:54
@@ -93,7 +126,7 @@ public static Cache<String, LookupFile> createCache(
}

private static int fileWeigh(String file, LookupFile lookupFile) {
return fileKibiBytes(lookupFile.localFile);
return lookupFile.isUsed() ? 0 : fileKibiBytes(lookupFile.localFile);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely don't understand why just marking a boolean can save disk space

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking the boolean is to make the weight to zero while this file is using. This could avoid the lookup file being evicted when putting it to a nearly full cache.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just:

LookupFile lookupFile = lookupFileCache.getIfPresent(file.fileName());
boolean newLookupFile = false;
if (lookupFile is null or lookupFile is closed) {
     lookupFile = createLookupFile();
     newLookupFile = true;
}
try {
     ..... use lookup file
} finally {
    if (newLookupFile) {
           lookupFileCache.put(file.fileName(), lookupFile);
    }
}


Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lookupFileCache.put(file.fileName(), newLookupFile); This still could evict this new create file out directly?

@Aitozi
Copy link
Contributor Author

Aitozi commented Aug 7, 2024

@JingsongLi Have updated to lookup the file before putting to cache

We may update to above 3.1.2 later to solve this strange evicting behavior according to this. (In test, it's better). But, the 3.x version requires JDK11 build.

@JingsongLi
Copy link
Contributor

@JingsongLi Have updated to lookup the file before putting to cache

We may update to above 3.1.2 later to solve this strange evicting behavior according to this. (In test, it's better). But, the 3.x version requires JDK11 build.

Thanks!

@JingsongLi
Copy link
Contributor

+1

@JingsongLi JingsongLi merged commit 3210a3e into apache:master Aug 7, 2024
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] While loop to cache build could lead to harmful IO operation
2 participants