[Backport 2.x] [Tiered caching] Integrating IndicesRequestCache with CacheService controlled by a feature flag

[sgup432]

Description

Backports #12533

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[sgup432]

@msfroh Can you help with this backport PR as well?

[github-actions]

Compatibility status:

Checks if related components are compatible with change fd9cd07

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❕ Gradle check result for 4be88c0: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.search.SearchWeightedRoutingIT.testShardRoutingWithNetworkDisruption_FailOpenEnabled
      1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=indices.get_field_mapping/20_missing_field/Return empty object if field doesn't exist, but index does}

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 94.73684% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 71.22%. Comparing base (0dd892c) to head (fd9cd07).
Report is 22 commits behind head on 2.x.

Files	Patch %	Lines
.../opensearch/common/cache/service/CacheService.java	50.00%	0 Missing and 1 partial ⚠️
...arch/common/cache/store/OpenSearchOnHeapCache.java	90.90%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #12600      +/-   ##
============================================
- Coverage     71.28%   71.22%   -0.06%     
- Complexity    60145    60179      +34     
============================================
  Files          4957     4958       +1     
  Lines        282799   282992     +193     
  Branches      41409    41436      +27     
============================================
- Hits         201591   201571      -20     
- Misses        64189    64391     +202     
- Partials      17019    17030      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for b4f2a8a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for fd9cd07: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.