Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] Support guava cache in CacheManager #4389

Merged
merged 3 commits into from
Oct 30, 2024

Conversation

FangYongs
Copy link
Contributor

Purpose

Linked issue: close #4388

Support guava cache type in CacheManager

Tests

Added test case FileBasedRandomInputViewTest and FileBasedBloomFilterTest

API and Format

no

Documentation

no

Copy link
Contributor

@Aitozi Aitozi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@JingsongLi
Copy link
Contributor

Hi @FangYongs @Aitozi , if you think guava cache is better, can you provide the benchmark result?

@mxdzs0612
Copy link
Contributor

mxdzs0612 commented Oct 29, 2024

There is still caffeine2 cache in paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java, which includes expiration by both time and weight, and it seems no compatibility now.

@JingsongLi
Copy link
Contributor

Hi @FangYongs @Aitozi , after #4401 , can you test caffiene 2 still have problem?

@FangYongs FangYongs force-pushed the support-guava-cache branch from 91708e7 to 6cb9cee Compare October 30, 2024 05:38
@FangYongs
Copy link
Contributor Author

@JingsongLi I have rebase #4401 and add benchmark for caffiene 2 and guava. When the cache get a value after it is removed before, the results show that in this case, guava will have higher performance than caffeine 2, and due to the issues of caffeine 2, the more accesses there are, the greater the performance gap.

Running benchmark: cache-benchmark
Running case: cache-CAFFEINE
Iteration 0 took 5032105 microseconds
Iteration 1 took 5026636 microseconds
Iteration 2 took 5024969 microseconds
Iteration 3 took 5025635 microseconds
Iteration 4 took 5030947 microseconds
Stopped after 5 iterations, 25140 ms
Running case: cache-GUAVA
Iteration 0 took 2014522 microseconds
Iteration 1 took 2018061 microseconds
Iteration 2 took 2012360 microseconds
Iteration 3 took 2015423 microseconds
Iteration 4 took 2016151 microseconds
Stopped after 5 iterations, 10076 ms

cache-benchmark: Best/Avg Time(ms) Row Rate(K/s) Per Row(ns) Relative

OPERATORTEST_cache-benchmark_cache-CAFFEINE 5025 / 5028 0.0 50249690.8 1.0X
OPERATORTEST_cache-benchmark_cache-GUAVA 2012 / 2015 0.0 20123608.3 2.5X

Copy link
Contributor

@JingsongLi JingsongLi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi JingsongLi merged commit 8b31057 into apache:master Oct 30, 2024
11 of 12 checks passed
hang8929201 pushed a commit to hang8929201/paimon that referenced this pull request Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Support guava cache in CacheManager
5 participants