Add a counter to node stat api to track shard going from idle to non-idle

[ruai0511]

Description

Shards automatically refresh every second, but when a shard doesn't receive search requests for over 30 seconds, it goes into an idle state to improve performance by suspending the implicit index refresh (More information on search idle feature here). However, this introduces a problem: After a shard does idle, the next search request must force a refresh to reflect the latest data. This extra step increases latency.

We want to monitor how often idle shards are reactivated. This PR introduces a counter called search_idle_waken_up_total and exports it in the node stat api.

Related Issues

Resolves #12678

Check List

• New functionality includes testing.
• All tests pass
• New functionality has been documented.
• New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 5d47d6b

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

✅ Gradle check result for 3a13164: SUCCESS

[ruai0511]

Good point. I'll address it in a later PR

I think it will make it easier if you should address it all in the same PR because you will need the same version checks for those APIs and it will make backporting a lot easier if you only have 1 PR to backport instead of 2.

Ah you're right about backport! Thanks I'll look into it

[github-actions]

❌ Gradle check result for 919e4e2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for ef646c6: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for c5ee63f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 8ec3c12: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 2d9bb3d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[jainankitk]

@ruai0511 - Thank you for this PR. I am wondering if we should also include whether refresh triggered or not as part of search request into coordinator slow logs. That will prevent us from guessing whether refresh happened or not causing the query to run slower.

[msfroh]

Do you want to add this metric to the tabular (_cat) APIs as well?

For example in server/src/main/java/org/opensearch/rest/action/cat/RestIndicesAction.java (and the corresponding _cat/nodes, _cat/shards, etc?)

This will probably require changes to the rest spec tests as well. You can use #9622 as a reference.

If you choose to add it to a _cat API (probably _cat/shards, since it's a shard-level metric), please don't add it to the default set of columns. While _node/stats is very verbose (and more verbose with every release), the default set of columns from _cat APIs tend to be limited to the most important values to get a summary of the given resource.

[github-actions]

❌ Gradle check result for 49ebf20: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for d2a686e: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 5172c09: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 5d47d6b: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.cluster.allocation.ClusterRerouteIT.testDelayWithALargeAmountOfShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[msfroh]

Thank, @ruai0511! Looks good!

I'll add the backport 2.x label, but we'll need to update the PR to change the version check for the new field.

Can you please prepare a PR to update the version check from 3.0 to 2.14?

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12768-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 f2cc3d8ea314b2d49a8b6ea5f57de5b6aff4faf9
# Push it to GitHub
git push --set-upstream origin backport/backport-12768-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12768-to-2.x.