[Tiered Caching] Make Indices Request Cache Stale Key Mgmt Threshold setting dynamic

[kiranprakash154]

Description

This PR includes the following changes -
Making Indices Request Cache Stale Key Mgmt Threshold setting dynamic.

#12625 Introduced Stale Key mgmt to Indices Request Cache.
In this we introduced the threshold setting as a static setting.
As an optimization we also stopped keeping track of stale keys when threshold is 0 when by default on-heap implementation of Tiered Cache was being used.

Now that we want to make this a dynamic setting, I have removed that check. That means, we keep track of staleness regardless of the threshold.

When the user updates the threshold setting, it would update the threshold field the cache cleaner looks at to determine if the cleanup is necessary or not.

The reason for going with this approach is because the cost of maintaining stale keys is minimal and this provides a seamless way for the cache cleaner to work when the thresholds are updated.

The memory implication of the map is the Integers being used, rest 2 items (ShardId and ReaderCacheKeyId) are already created and we just use them by reference.

A rough calculation of the memory implication.

We recommend users max of 1k shards, with a default refresh rate of 1 sec and each second having an entry of 4B, and the cache cleaner by default runs every minute, the calculation will turn out to be
1_000 x 60 x 4B = 240KB.

Related Issues

Resolves #12540

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change bc00536

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/sql.git]

[github-actions]

❌ Gradle check result for 09ca73a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 56386ca: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 1e4927b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 739e791: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c6a0036: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[kiranprakash154]

❌ Gradle check result for c6a0036: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Known Flaky Test - #7755

[github-actions]

✅ Gradle check result for bc00536: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 83.33333% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 71.31%. Comparing base (b15cb0c) to head (bc00536).
Report is 107 commits behind head on main.

Files	Patch %	Lines
...va/org/opensearch/indices/IndicesRequestCache.java	83.33%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12975      +/-   ##
============================================
- Coverage     71.42%   71.31%   -0.11%     
- Complexity    59978    60264     +286     
============================================
  Files          4985     5021      +36     
  Lines        282275   284076    +1801     
  Branches      40946    41151     +205     
============================================
+ Hits         201603   202600     +997     
- Misses        63999    64659     +660     
- Partials      16673    16817     +144     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sgup432]

This one seems to be stale. Please see #12941 for the updated change.

[kiranprakash154]

This can be closed since #12941 contains the changes planned in this PR