Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history 
commit 93a3132
Author: Peter Alfonsi <[email protected]>
Date:   Wed Sep 4 11:44:03 2024 -0700

    fix broken ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

commit fd92c5a
Author: Marc Handalian <[email protected]>
Date:   Thu Jan 11 06:41:55 2024 -0800

    Update runTask to optionally install plugins (opensearch-project#11844)

    Signed-off-by: Marc Handalian <[email protected]>

commit 4d78661
Author: Peter Alfonsi <[email protected]>
Date:   Tue Sep 3 16:02:31 2024 -0700

    Misc test fixes

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 70a6935
Author: Peter Alfonsi <[email protected]>
Date:   Fri Aug 23 10:42:05 2024 -0700

    [Bugfix] Fixes IRC NPE bug for timed-out cacheable queries (opensearch-project#15327)

    * Fix IRC timeout bug

    Signed-off-by: Peter Alfonsi <[email protected]>

    * addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Add UT for test coverage

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * tweak imports in new UT

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit f9943f8
Author: Sagar <[email protected]>
Date:   Wed Jul 17 19:02:05 2024 -0700

    Clear ehcache disk cache files during initialization (opensearch-project#14738)

    * Clear ehcache disk cache files during initialization

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding UT to fix line coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding more Uts for better line coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Throwing exception in case we fail to clear cache files during startup

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding more UTs

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a UT for more coverage

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle build

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Update ehcache disk cache close() logic

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit fc2007e
Author: Peter Alfonsi <[email protected]>
Date:   Mon Jul 1 15:54:39 2024 -0700

    [Bugfix] Fix ICacheKeySerializerTests flakiness (opensearch-project#14564)

    * Fix testInvalidInput flakiness

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun security check

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 1ef62e2
Author: Sagar <[email protected]>
Date:   Wed Jun 26 11:14:42 2024 -0700

    Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently (opensearch-project#14550)

    * Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 29ec9d6
Author: Sagar <[email protected]>
Date:   Tue Jun 25 12:04:17 2024 -0700

    [Tiered Caching]  Moving query recomputation logic outside of write lock (opensearch-project#14187)

    * Moving query recompute out of write lock

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * [Tiered Caching] Moving query recomputation logic outside of write lock

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding java doc for the completable map

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Changes to call future handler only once per key

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing spotless check

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle fail

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments to refactor unit test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * minor UT refactor

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 8347779
Author: Peter Alfonsi <[email protected]>
Date:   Tue Jun 25 13:26:54 2024 -0700

    Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462)

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 40f868d
Author: Kiran Prakash <[email protected]>
Date:   Thu Jun 20 14:17:23 2024 -0700

    Update IndicesRequestCacheCleanupIT.java (opensearch-project#14478)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 094a9f2
Author: Kiran Prakash <[email protected]>
Date:   Wed Jun 19 15:36:59 2024 -0700

    Fix Indices Request Cache ITs from flaking and timing out (opensearch-project#14369)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 3a110a8
Author: Kiran Prakash <[email protected]>
Date:   Tue Jun 18 10:18:25 2024 -0700

    [Tiered Cache] Use ConcurrentHashMap explicitly in IndicesRequestCache (opensearch-project#14409)

    Signed-off-by: Kiran Prakash <[email protected]>

commit bcf0f59
Author: Peter Alfonsi <[email protected]>
Date:   Fri Jun 14 13:20:51 2024 -0700

    [Bugfix] Fix TieredSpilloverCache flaky tests (opensearch-project#14333)

    * Fix flaky TSC stats tests

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed andrross's comment

    Signed-off-by: Peter Alfonsi <[email protected]>

    * fix forbidden API

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit a222a03
Author: Kiran Prakash <[email protected]>
Date:   Wed Jun 12 18:30:06 2024 -0700

    [Tiered Caching] [Bug Fix] Use concurrentMap instead of HashMap to fix Concurrent Modification Exception (opensearch-project#14221)

    * use concurrentmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert feature flags

    Signed-off-by: Kiran Prakash <[email protected]>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <[email protected]>

    * use concurrentmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert feature flags

    Signed-off-by: Kiran Prakash <[email protected]>

    * changelog to releaselog

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert the test removal

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert the conflict resolutions

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f85cd00
Author: Sagar <[email protected]>
Date:   Wed Jun 12 08:22:38 2024 -0700

    Fix ShardNotFoundException during request cache clean up (opensearch-project#14219)

    * Fix for ShardNotFoundException during request cache clean up

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fix forbidden gradle check

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 406ae78
Author: Sagar <[email protected]>
Date:   Thu Jun 6 15:02:00 2024 -0700

    [Caching] Move cache removal notifications outside lru lock (opensearch-project#14017)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit c5fd7fe
Author: Peter Alfonsi <[email protected]>
Date:   Tue Sep 3 14:38:26 2024 -0700

    Change IRC UT to correctly create shard

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 9c4d4af
Author: Peter Alfonsi <[email protected]>
Date:   Tue Jun 4 16:09:18 2024 -0700

    [Tiered Caching] Additional ITs for cache stats  (opensearch-project#13655)

    * Adds cache clear IT

    Signed-off-by: Peter Alfonsi <[email protected]>

    Cleaned up logic for cache stats ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

    Adds more tests around tiered spillover cache

    Signed-off-by: Peter Alfonsi <[email protected]>

    Fixed cache stats behavior for overall /_nodes/stats call

    Signed-off-by: Peter Alfonsi <[email protected]>

    cleanup

    Signed-off-by: Peter Alfonsi <[email protected]>

    Fixed folder structure

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Sagar's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Ankit's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    Break horrifyingly long test case into many shorter cases

    Signed-off-by: Peter Alfonsi <[email protected]>

    Added unsupported operation exception to TSC stats holder incrementEvictions()

    Signed-off-by: Peter Alfonsi <[email protected]>

    Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun assemble

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit be97e90
Author: Peter Alfonsi <[email protected]>
Date:   Thu May 23 12:02:22 2024 -0700

    [Bugfix] [Tiered Caching] Fixes issues when integrating tiered cache with disk cache (opensearch-project#13784)

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 60ee715
Author: Kiran Prakash <[email protected]>
Date:   Mon May 20 12:52:03 2024 -0700

    Update IndicesRequestCacheIT.java (opensearch-project#13678)

    Signed-off-by: Kiran Prakash <[email protected]>

commit 51f6c1b
Author: Sagar <[email protected]>
Date:   Wed May 8 09:15:36 2024 -0700

    Fix IndicesRequestCache clean up logic (opensearch-project#13597)

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Co-authored-by: Sagar Upadhyaya <[email protected]>

commit 49e2701
Author: Sagar <[email protected]>
Date:   Mon May 6 17:25:45 2024 -0700

    Fix negative requestStats memory_size issue (opensearch-project#13553)

    This solves the bug where RequestStats memory_size metric was going negative in certain scenarios
    as reported in the issue. It turns out that the issue occurs when an indexShard is deleted and then
    reallocated on the same node. So whenever stale entries from older shard are deleted, those are
    accounted for the new shard which has the same shardId.

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit a5c5675
Author: Peter Alfonsi <[email protected]>
Date:   Thu May 2 17:29:07 2024 -0700

    [Tiered Caching] Adds stats implementation for TieredSpilloverCache (opensearch-project#13236)

    Stats rework part 4 of 4
    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 46944cb
Author: peteralfonsi <[email protected]>
Date:   Tue Apr 30 10:59:03 2024 -0700

    Fix flaky test CacheStatsAPIIndicesRequestCacheIT.testNullLevels() (opensearch-project#13457)

    * Fix flaky test

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Initialize CommonStatsFlags with empty array for levels

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Fixes tests using incorrect null levels

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 71fcad4
Author: Kiran Prakash <[email protected]>
Date:   Mon Apr 29 18:41:33 2024 -0700

    Fix Flaky test IndicesRequestCacheIT.testStaleKeysCleanupWithMultipleIndices (opensearch-project#13453)

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f60339f
Author: peteralfonsi <[email protected]>
Date:   Mon Apr 29 22:37:05 2024 -0700

    [Tiered Caching] Bump versions for serialization in new cache stats API (opensearch-project#13460)

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 88718a2
Author: peteralfonsi <[email protected]>
Date:   Mon Apr 29 14:47:52 2024 -0700

    [Tiered Caching] Expose new cache stats API (opensearch-project#13237)

    Step 3 out of 4

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 5e97082
Author: peteralfonsi <[email protected]>
Date:   Sun Apr 28 08:15:18 2024 -0700

    [Tiered Caching] Gate CacheStatsHolder logic behind FeatureFlags.PLUGGABLE_CACHE setting (opensearch-project#13238)

    Stats rework step 2 of 4

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit ed881f7
Author: Kiran Prakash <[email protected]>
Date:   Fri Apr 26 11:52:31 2024 -0700

    [Tiered Caching] Make Indices Request Cache Stale Key Mgmt Threshold setting dynamic (opensearch-project#12941)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update ClusterSettings.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * some refactoring

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address existing tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * UTs

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * ITs

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * refactor

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * address code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * address code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * resolve conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheIT.java

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit 424ccc9
Author: Sagar <[email protected]>
Date:   Thu Apr 25 22:09:42 2024 -0700

    [Tiered Caching] Expose a dynamic setting to disable/enable disk cache (opensearch-project#13373)

    * [Tiered Caching] Expose a dynamic setting to disable/enable disk cache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Putting tiered cache settings behind feature flag

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing Sorabh's comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Putting new setting behind feature flag

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit acbbb2f
Author: Kiran Prakash <[email protected]>
Date:   Wed Apr 24 14:50:39 2024 -0700

    [Tiered Caching] Bug fix for IndicesRequestCache StaleKey management (opensearch-project#13070)

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * code comments only

    Signed-off-by: Kiran Prakash <[email protected]>

    * docs changes

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * revert catching AlreadyClosedException

    Signed-off-by: Kiran Prakash <[email protected]>

    * assert

    Signed-off-by: Kiran Prakash <[email protected]>

    * conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCacheTests.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update IndicesRequestCache.java

    Signed-off-by: Kiran Prakash <[email protected]>

    * address conflicts

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless apply

    Signed-off-by: Kiran Prakash <[email protected]>

    * address comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * update code comments

    Signed-off-by: Kiran Prakash <[email protected]>

    * address bug & add tests

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit f1d2e72
Author: Sagar <[email protected]>
Date:   Mon Apr 15 10:43:22 2024 -0800

    [Tiered Caching] Ehcache Disk cache IT  (opensearch-project#12904)

    * Ehcache IT tests

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding some logs to print key/value size

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Add ehcache related invalidation IT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Remvoing unnecessary IndicesRequestCache IT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Indentation fix

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added tests around expiration time and invalidation

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit b0de406
Author: peteralfonsi <[email protected]>
Date:   Fri Apr 12 16:09:06 2024 -0700

    [Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers (opensearch-project#12531)

    As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains
    dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the
    CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by
    these dimensions.

    Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the
    TieredSpilloverCache will be in a followup PR.

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 40a8d34
Author: Sagar <[email protected]>
Date:   Thu Apr 11 14:49:03 2024 -0800

    [Tiered Caching] Make took time policy dynamic and add additional integ tests (opensearch-project#13063)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit 6f88777
Author: Sagar <[email protected]>
Date:   Tue Mar 19 19:41:59 2024 -0700

    Fixing ehcache flaky test (opensearch-project#12764)

    * Fixing ehcache flaky test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding a ehcache issue reference for thread leak issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Updating comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit f69c089
Author: Sagar <[email protected]>
Date:   Mon Mar 18 22:20:52 2024 -0700

    [Tiered Caching] Clear up disk cache(ehcache) files during node shutdown (opensearch-project#12734)

    * Adding logic to clear up the disk cache files during close()
    * Adding logic to update entries count after invalidateAll()
    * Removing unneeded system log statement
    * Added comment in test for readability
    * Fixing issue where we were sending compacted byte[] array to ehcache but calculating size with padded byte[]

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit bf6488d
Author: Sagar <[email protected]>
Date:   Mon Mar 18 22:17:14 2024 -0700

    [Tiered Caching] Fix test testComputeIfAbsentWithFactoryBasedCacheCreation (opensearch-project#12700)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 0fc1f40
Author: peteralfonsi <[email protected]>
Date:   Mon Mar 18 13:39:35 2024 -0700

    [Tiered Caching] Serializers for ehcache (opensearch-project#12709)

    Adds serializers and integrates them into ehcache disk cache

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit a45be0f
Author: Kiran Prakash <[email protected]>
Date:   Sat Mar 16 17:12:27 2024 -0700

    [Tiered Caching] Indices Request cache stalekey management (opensearch-project#12625)

    * Introduce IndicesRequestCacheCleanupManager

    Signed-off-by: Kiran Prakash <[email protected]>

    * using cleanup mgr to enqueue cleanups

    Signed-off-by: Kiran Prakash <[email protected]>

    * readability improvements

    Signed-off-by: Kiran Prakash <[email protected]>

    * update updateCleanupKeyToCountMap on new cache entry

    Signed-off-by: Kiran Prakash <[email protected]>

    * create IndicesRequestCacheCleanupManager & settings and validators

    Signed-off-by: Kiran Prakash <[email protected]>

    * Add IRC specific cache cleaner and remove from IndicesService

    Signed-off-by: Kiran Prakash <[email protected]>

    * Unit Tests

    Signed-off-by: Kiran Prakash <[email protected]>

    * Update CHANGELOG.md

    Signed-off-by: Kiran Prakash <[email protected]>

    * move cachecleaner inside mgr

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove processCleanupKeys

    Signed-off-by: Kiran Prakash <[email protected]>

    * minor cleanups

    Signed-off-by: Kiran Prakash <[email protected]>

    * add updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove locks and make all methods synchronized

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotless

    Signed-off-by: Kiran Prakash <[email protected]>

    * updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * Testing

    Signed-off-by: Kiran Prakash <[email protected]>

    * add Reschedule back to indices service

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename updateStaleKeysCount to incrementStaleKeysCount

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename getStaleKeysCountForTesting to getStaleKeysCount

    Signed-off-by: Kiran Prakash <[email protected]>

    * rename threshold to stalenessThreshold

    Signed-off-by: Kiran Prakash <[email protected]>

    * check for cleanupKey.entity == null

    Signed-off-by: Kiran Prakash <[email protected]>

    * use computeIfPresent with keycountmap

    Signed-off-by: Kiran Prakash <[email protected]>

    * log both staleKeysInCache & Staleness in debug logs

    Signed-off-by: Kiran Prakash <[email protected]>

    * Use HashMap instead of ConcurrentMap

    Signed-off-by: Kiran Prakash <[email protected]>

    * Address b/w compatibility

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove synchronized for updateCleanupKeyToCountMapOnCacheEviction

    Signed-off-by: Kiran Prakash <[email protected]>

    * make cleanCache synchronized

    Signed-off-by: Kiran Prakash <[email protected]>

    * remove shouldRemoveKey

    Signed-off-by: Kiran Prakash <[email protected]>

    * spotlessApply

    Signed-off-by: Kiran Prakash <[email protected]>

    ---------

    Signed-off-by: Kiran Prakash <[email protected]>

commit bad412c
Author: Sagar <[email protected]>
Date:   Fri Mar 15 16:53:54 2024 -0700

    [Tiered caching] Supporting removal function on EhcacheDiskCache iterator (opensearch-project#12653)

    * [Tiered caching] Supporting removal function on EhcacheDiskCache iterator

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Minor refactoring in unit test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit 15c4afa
Author: peteralfonsi <[email protected]>
Date:   Fri Mar 15 15:36:22 2024 -0700

    [Tiered Caching] Cache tier policies (opensearch-project#12542)

    * Adds policy interface and took time policy impl

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Changes IndicesService to write a CachePolicyInfoWrapper before the QSR

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Moved took time logic from QSR to IndicesService

    Signed-off-by: Peter Alfonsi <[email protected]>

    * spotlessApply

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed ansjcy's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Partial rebase on most recent changes

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Integrated policies with new TSC changes

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Reverted unintended change to idea/vcs.xml

    Signed-off-by: Peter Alfonsi <[email protected]>

    * javadocs

    Signed-off-by: Peter Alfonsi <[email protected]>

    * github actions

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Set default threshold value to 10 ms

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's second round of comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Set cachedQueryParser in IRC

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Addressed Sorabh's comments besides dynamic setting

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Removed dynamic setting, misc comments

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added changelog entry

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added missing javadoc

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Fixed failed gradle run

    Signed-off-by: Peter Alfonsi <[email protected]>

    * Added setting validation test

    Signed-off-by: Peter Alfonsi <[email protected]>

    * rerun gradle for flaky IT

    Signed-off-by: Peter Alfonsi <[email protected]>

    * javadocs

    Signed-off-by: Peter Alfonsi <[email protected]>

    ---------

    Signed-off-by: Peter Alfonsi <[email protected]>
    Co-authored-by: Peter Alfonsi <[email protected]>

commit 45fce15
Author: Sagar <[email protected]>
Date:   Thu Mar 14 13:00:24 2024 -0700

    [Tiered Caching] Fixing flaky tiered cache test (opensearch-project#12650)

    * Fixing flaky tiered cache test

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unnecessary comment

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unused variable

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>

commit bb66699
Author: Peter Alfonsi <[email protected]>
Date:   Fri Aug 30 11:36:05 2024 -0700

    Attempt to fix broken ITs

    Signed-off-by: Peter Alfonsi <[email protected]>

commit 58170bf
Author: Sagar <[email protected]>
Date:   Mon Mar 11 14:12:55 2024 -0700

    [Tiered caching] Integrating IndicesRequestCache with CacheService controlled by a feature flag (opensearch-project#12533)

    * Adding changelog
    * Fixing gradle build issue
    * Fixing CacheService test
    * Adding UT in IndicesRequestCache with feature flag for more coverage
    * Updating changelog and renaming feature flag setting
    * Moving feature flag setting handling logic to CacheService by maintaining backward compatibility
    * Fixing broken UTs

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit 80d2130
Author: Sagar <[email protected]>
Date:   Fri Mar 1 11:10:28 2024 -0800

    [Tiered caching] Introducing cache plugins and exposing Ehcache as one of the pluggable disk cache option (opensearch-project#11874)

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

commit f372c2c
Author: Sagar <[email protected]>
Date:   Wed Jan 10 17:06:29 2024 -0800

    [Tiered Caching] Enable serialization of IndicesRequestCache.Key (opensearch-project#10275)

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>
    Co-authored-by: Kiran Prakash <[email protected]>

commit b880c0b
Author: Sagar <[email protected]>
Date:   Tue Jan 9 10:57:52 2024 -0800

    [Tiered caching] Framework changes (opensearch-project#10753)

    * [Tiered caching] Framework changes

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added javadoc for new files/packages

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Added changelog

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing javadoc warnings

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Addressing additional minor comments

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Moving non null check to builder for OS onHeapCache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding package-info for new packages

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing service and adding different cache interfaces along with event listener support

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing gradle missingDoc issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Changing listener logic, removing tiered cache integration with IRC

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding opensearch.internal tag for LoadAwareCacheLoader

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Fixing thread safety issue

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Remove compute function and event listener logic change for TieredCache

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Making Cache.compute function private

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding javadoc and more test for cache.put

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Adding write locks to refresh API as well

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing unwanted EventType class and refactoring one UT

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    * Removing TieredCache interface

    Signed-off-by: Sagar Upadhyaya <[email protected]>

    ---------

    Signed-off-by: Sagar Upadhyaya <[email protected]>
    Signed-off-by: Sagar <[email protected]>

Signed-off-by: Peter Alfonsi <[email protected]>
  • Loading branch information
Peter Alfonsi committed Sep 4, 2024
1 parent a5cc554 commit 60f3710
Show file tree
Hide file tree
Showing 119 changed files with 15,642 additions and 529 deletions.
8 changes: 7 additions & 1 deletion DEVELOPER_GUIDE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,12 @@ Run OpenSearch using `gradlew run`.
./gradlew run
```

[Plugins](plugins/) may be installed by passing a `-PinstalledPlugins` property:

```bash
./gradlew run -PinstalledPlugins="['plugin1', 'plugin2']"
```

That will build OpenSearch and start it, writing its log above Gradle's status message. We log a lot of stuff on startup, specifically these lines tell you that OpenSearch is ready.

```
Expand Down Expand Up @@ -578,7 +584,7 @@ explicitly marked by an annotation should not be extended by external implementa
any time. The `@DeprecatedApi` annotation could also be added to any classes annotated with `@PublicApi` (or documented as `@opensearch.api`) or their methods that
are either changed (with replacement) or planned to be removed across major versions.

The APIs which are designated to be public but have not been stabilized yet should be marked with `@ExperimentalApi` (or documented as `@opensearch.experimental`)
The APIs which are designated to be public but have not been stabilized yet should be marked with `@ExperimentalApi` (or documented as `@opensearch.experimental`)
annotation. The presence of this annotation signals that API may change at any time (major, minor or even patch releases). In general, the classes annotated with
`@PublicApi` may expose other classes or methods annotated with `@ExperimentalApi`, in such cases the backward compatibility guarantees would not apply to latter
(see please [Experimental Development](#experimental-development) for more details).
Expand Down
3 changes: 3 additions & 0 deletions distribution/src/config/opensearch.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,6 @@ ${path.logs}
# index searcher threadpool.
#
#opensearch.experimental.feature.concurrent_segment_search.enabled: false
# Gates the functionality of enabling Opensearch to use pluggable caches with respective store names via setting.
#
#opensearch.experimental.feature.pluggable.caching.enabled: false
6 changes: 6 additions & 0 deletions gradle/run.gradle
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,12 @@ apply plugin: 'opensearch.testclusters'
testClusters {
runTask {
testDistribution = 'archive'
if (findProperty("installedPlugins")) {
installedPlugins = Eval.me(installedPlugins)
for (String p : installedPlugins) {
plugin('plugins:'.concat(p))
}
}
}
}

Expand Down
5 changes: 5 additions & 0 deletions libs/core/src/main/java/org/opensearch/core/common/bytes/BytesReference.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,11 @@ static byte[] toBytes(BytesReference reference) {
return ArrayUtil.copyOfSubArray(bytesRef.bytes, bytesRef.offset, bytesRef.offset + bytesRef.length);
}

static byte[] toBytesWithoutCompact(BytesReference reference) {
final BytesRef bytesRef = reference.toBytesRef();
return bytesRef.bytes;
}

/**
* Returns an array of byte buffers from the given BytesReference.
*/
Expand Down
40 changes: 39 additions & 1 deletion libs/core/src/main/java/org/opensearch/core/common/io/stream/StreamInput.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Comparator;
import java.util.Date;
import java.util.EnumSet;
import java.util.HashMap;
Expand All @@ -89,6 +90,8 @@
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;
import java.util.concurrent.TimeUnit;
import java.util.function.IntFunction;

Expand Down Expand Up @@ -641,12 +644,47 @@ public <K, V> Map<K, V> readMap(Writeable.Reader<K> keyReader, Writeable.Reader<
return Collections.emptyMap();
}
Map<K, V> map = new HashMap<>(size);
readIntoMap(keyReader, valueReader, map, size);
return map;
}

/**
* Read a serialized map into a SortedMap using the default ordering for the keys. If the result is empty it might be immutable.
*/
public <K extends Comparable<K>, V> SortedMap<K, V> readOrderedMap(Writeable.Reader<K> keyReader, Writeable.Reader<V> valueReader)
throws IOException {
return readOrderedMap(keyReader, valueReader, null);
}

/**
* Read a serialized map into a SortedMap, specifying a Comparator for the keys. If the result is empty it might be immutable.
*/
public <K extends Comparable<K>, V> SortedMap<K, V> readOrderedMap(
Writeable.Reader<K> keyReader,
Writeable.Reader<V> valueReader,
@Nullable Comparator<K> keyComparator
) throws IOException {
int size = readArraySize();
if (size == 0) {
return Collections.emptySortedMap();
}
SortedMap<K, V> sortedMap;
if (keyComparator == null) {
sortedMap = new TreeMap<>();
} else {
sortedMap = new TreeMap<>(keyComparator);
}
readIntoMap(keyReader, valueReader, sortedMap, size);
return sortedMap;
}

private <K, V> void readIntoMap(Writeable.Reader<K> keyReader, Writeable.Reader<V> valueReader, Map<K, V> map, int size)
throws IOException {
for (int i = 0; i < size; i++) {
K key = keyReader.read(this);
V value = valueReader.read(this);
map.put(key, value);
}
return map;
}

/**
Expand Down
7 changes: 7 additions & 0 deletions libs/core/src/main/java/org/opensearch/core/index/shard/ShardId.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@

package org.opensearch.core.index.shard;

import org.apache.lucene.util.RamUsageEstimator;
import org.opensearch.common.annotation.PublicApi;
import org.opensearch.core.common.Strings;
import org.opensearch.core.common.io.stream.StreamInput;
Expand All @@ -55,6 +56,8 @@ public class ShardId implements Comparable<ShardId>, ToXContentFragment, Writeab
private final int shardId;
private final int hashCode;

private final static long BASE_RAM_BYTES_USED = RamUsageEstimator.shallowSizeOfInstance(ShardId.class);

/**
* Constructs a new shard id.
* @param index the index name
Expand Down Expand Up @@ -88,6 +91,10 @@ public ShardId(StreamInput in) throws IOException {
hashCode = computeHashCode();
}

public long getBaseRamBytesUsed() {
return BASE_RAM_BYTES_USED;
}

/**
* Writes this shard id to a stream.
* @param out the stream to write to
Expand Down
19 changes: 19 additions & 0 deletions modules/cache-common/build.gradle
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
/*
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

apply plugin: 'opensearch.internal-cluster-test'

opensearchplugin {
description 'Module for caches which are optional and do not require additional security permission'
classname 'org.opensearch.cache.common.tier.TieredSpilloverCachePlugin'
}

test {
// TODO: Adding permission in plugin-security.policy doesn't seem to work.
systemProperty 'tests.security.manager', 'false'
}
Loading

0 comments on commit 60f3710

Please sign in to comment.