Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504

amkhar · 2024-03-01T09:24:28Z

Description

Add a new ShardBatchCache which will handle storing the responses of transport actions like TransportNodesListGatewayStartedShardsBatch or the transport action for replica shards being pushed via #8957.

A new cache class is being written as storing strategy is changed from ShardCache (written #12441).
ShardCache - stores the responses of transport actions as it is in NodeEntry class

https://github.com/opensearch-project/OpenSearch/pull/12441/files#diff-e2790d7ec0cf48617430d2352e7142c297eaea172fd6e2d34969e60ddf7f9a68R73

https://github.com/amkhar/OpenSearch/blob/3125b948029609f354d3153f8ca6391638daefc7/server/src/main/java/org/opensearch/gateway/AsyncShardFetch.java#L468-L475

    static class NodeEntry<T extends BaseNodeResponse> extends BaseShardCache.BaseNodeEntry {
        
        @Nullable
        private T value;

But the response of Transport actions like TransportNodesListGatewayStartedShardsBatch are a map

Map<ShardId, NodeGatewayStartedShard>

Storing this response as it is will create repetition for ShardId object, as the responses needs to be stored for every node.
Map<NodeId, Map<ShardId, NodeGatewayStartedShard>>

To solve the problem, storing the NodeGatewayStartedShard in an array and keeping ShardId to arrayIndex in a map

private final NodeGatewayStartedShard[] shardData;
private final Map<ShardId, Integer> shardIdKey = new HashMap<>();

Using these two data structures, repetition is avoided and array index is used for putting and getting the data from cache.
Failure handling is also different from normal ShardCache, as this is a batch of ShardIds. Based on unknown failures for a ShardId, we need to retry that shard only in next run of reroute. And allocate all other shards of the batch.

Related Issues

Resolves ##12248
This PR is dependent on
#12441
#8356

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
~~Commit changes are listed out in CHANGELOG.md file (See: Changelog)~~
~~Public documentation issue/PR created~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ch of shards Signed-off-by: Aman Khare <[email protected]>

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-03-01T09:27:26Z

Compatibility status:

Checks if related components are compatible with change 08763ed

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2024-03-01T09:30:43Z

❌ Gradle check result for bcbc00a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-03-05T18:15:12Z

❌ Gradle check result for 644d908: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

server/src/main/java/org/opensearch/gateway/TransportNodesListGatewayStartedShardsBatch.java

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

server/src/main/java/org/opensearch/gateway/TransportNodesListGatewayStartedShardsBatch.java

server/src/main/java/org/opensearch/indices/store/ShardAttributes.java

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-04-02T15:42:25Z

❌ Gradle check result for ba6cbb4: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-04-04T05:23:18Z

✅ Gradle check result for c0ed643: SUCCESS

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-04-08T09:19:45Z

❌ Gradle check result for d61477f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

amkhar · 2024-04-08T09:45:26Z

❌ Gradle check result for d61477f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Flaky test : #12197

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

server/src/main/java/org/opensearch/gateway/TransportNodesGatewayStartedShardHelper.java

server/src/main/java/org/opensearch/gateway/TransportNodesListGatewayStartedShardsBatch.java

server/src/test/java/org/opensearch/gateway/ShardBatchCacheTests.java

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-04-09T09:05:55Z

❕ Gradle check result for 65229fc: UNSTABLE

TEST FAILURES:

      2 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java

Bukhtawar

Thanks @amkhar too many explicit type conversions and reflection isn't usually the best way to get around structuring classes

server/src/main/java/org/opensearch/gateway/ShardBatchResponseFactory.java

Signed-off-by: Aman Khare <[email protected]>

github-actions · 2024-04-10T06:37:05Z

❌ Gradle check result for 08763ed: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-04-11T07:22:16Z

❕ Gradle check result for 08763ed: UNSTABLE

TEST FAILURES:

      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

amkhar · 2024-04-11T08:35:42Z

❕ Gradle check result for 08763ed: UNSTABLE

TEST FAILURES:
      1 org.opensearch.http.SearchRestCancellationIT.testAutomaticCancellationDuringFetchPhase
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Flaky test : #5426

…tartedShardsBatch (#12504) Signed-off-by: Aman Khare <[email protected]> (cherry picked from commit 7103e56) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…tartedShardsBatch (#12504) (#13156) (cherry picked from commit 7103e56) Signed-off-by: Aman Khare <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…tartedShardsBatch (opensearch-project#12504) Signed-off-by: Aman Khare <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

Aman Khare added 5 commits July 27, 2023 14:07

Add PrimaryShardBatchAllocator to take allocation decisions for a bat…

e87bfdd

…ch of shards Signed-off-by: Aman Khare <[email protected]>

Add unit tests and refactor PrimaryShardBatchAllocator

6c38303

Signed-off-by: Aman Khare <[email protected]>

Throw exception for single shard calls

1a68940

Signed-off-by: Aman Khare <[email protected]>

Modify according to transport PRs.

dc7eb43

Signed-off-by: Aman Khare <[email protected]>

Use List instead of Set

e0c4943

Signed-off-by: Aman Khare <[email protected]>

amkhar mentioned this pull request Mar 1, 2024

[META] Cluster Manager Async Shard Fetch Revamp #8098

Open

13 tasks

amkhar marked this pull request as ready for review March 6, 2024 05:36

amkhar requested review from abbashus, adnapibar, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kartg, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, ryanbogan and sachinpkale as code owners March 6, 2024 05:36

shwetathareja reviewed Mar 19, 2024

View reviewed changes

amkhar mentioned this pull request Mar 20, 2024

[Optimization] Reduce ShardId key size in AsyncShardFetch Revamp strategy #12010

Closed

shwetathareja reviewed Apr 1, 2024

View reviewed changes

Use single exception parameter, modify exception handling in batch mode

ba6cbb4

Signed-off-by: Aman Khare <[email protected]>

Aman Khare added 2 commits April 4, 2024 10:04

Add factory class to create new response objects

4246256

Signed-off-by: Aman Khare <[email protected]>

Merge remote-tracking branch 'origin/main' into shard-batch-cache

c0ed643

Signed-off-by: Aman Khare <[email protected]>

Save non empty response in else condition

d61477f

Signed-off-by: Aman Khare <[email protected]>

shwetathareja reviewed Apr 8, 2024

View reviewed changes

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java Show resolved Hide resolved

Cleanup java doc and review comments changes

65229fc

Signed-off-by: Aman Khare <[email protected]>

Bukhtawar reviewed Apr 9, 2024

View reviewed changes

server/src/main/java/org/opensearch/gateway/AsyncShardBatchFetch.java Show resolved Hide resolved

Bukhtawar reviewed Apr 9, 2024

View reviewed changes

server/src/main/java/org/opensearch/gateway/ShardBatchResponseFactory.java Show resolved Hide resolved

Mark clearShard method as synchronized

08763ed

Signed-off-by: Aman Khare <[email protected]>

amkhar mentioned this pull request Apr 10, 2024

[Restructure] Remove type casting in ShardBatchResponseFactory class #13146

Open

shwetathareja approved these changes Apr 11, 2024

View reviewed changes

shwetathareja merged commit 7103e56 into opensearch-project:main Apr 11, 2024
37 of 46 checks passed

shwetathareja added the backport 2.x Backport to 2.x branch label Apr 11, 2024

opensearch-trigger-bot bot mentioned this pull request Apr 11, 2024

[Backport 2.x] Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #13156

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504

Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504

amkhar commented Mar 1, 2024 •

edited

Loading

github-actions bot commented Mar 1, 2024 •

edited

Loading

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 5, 2024

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 8, 2024

amkhar commented Apr 8, 2024

github-actions bot commented Apr 9, 2024

Bukhtawar left a comment

github-actions bot commented Apr 10, 2024

github-actions bot commented Apr 11, 2024

amkhar commented Apr 11, 2024

Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504

Add ShardBatchCache to support caching for TransportNodesListGatewayStartedShardsBatch #12504

Conversation

amkhar commented Mar 1, 2024 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Mar 1, 2024 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Mar 1, 2024

github-actions bot commented Mar 5, 2024

github-actions bot commented Apr 2, 2024

github-actions bot commented Apr 4, 2024

github-actions bot commented Apr 8, 2024

amkhar commented Apr 8, 2024

github-actions bot commented Apr 9, 2024

Bukhtawar left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 10, 2024

github-actions bot commented Apr 11, 2024

amkhar commented Apr 11, 2024

amkhar commented Mar 1, 2024 •

edited

Loading

github-actions bot commented Mar 1, 2024 •

edited

Loading