[Remote Store] Add remote segment transfer stats on NodesStats API #9168

shourya035 · 2023-08-08T13:35:04Z

Description

Adding remote segment transfer stats on the NodesStats API output. As of now, we are including the following metrics:

total_uploads.started_bytes: Amount of segment payload attempted to be uploaded to the remote store
total_uploads.succeeded_bytes: Amount of segment payload successfully uploaded to the remote store
total_uploads.failed_bytes: Amount of segment payload failed to be uploaded to the remote store
max_refresh_time_lag_in_millis: Max refresh lag in milliseconds across all the shards on the node
max_refresh_size_lag_in_bytes: Max refresh lag in size across all the shards on the node
total_downloads.started_bytes: Amount of segment payload attempted to be downloaded from the remote store
total_downloads.succeeded_bytes: Amount of segment payload successfully downloaded from the remote store
total_downloads.failed_bytes: Amount of segment payload failed to be downloaded from the remote store

These are added to the indices.segment section of the NodesStats API output

~ ❯ curl --silent 'http://localhost:9200/_nodes/opensearch-node*/stats/indices/segments' | jq
{
  "_nodes": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "cluster_name": "opensearch-cluster",
  "nodes": {
    "yKBw3ByYSd6KZnx_qOb9NA": {
      "timestamp": 1691500774603,
      "name": "opensearch-node2",
      "transport_address": "172.18.0.3:9300",
      "host": "172.18.0.3",
      "ip": "172.18.0.3:9300",
      "roles": [
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "indices": {
        "segments": {
          "count": 3,
          "memory_in_bytes": 0,
          "terms_memory_in_bytes": 0,
          "stored_fields_memory_in_bytes": 0,
          "term_vectors_memory_in_bytes": 0,
          "norms_memory_in_bytes": 0,
          "points_memory_in_bytes": 0,
          "doc_values_memory_in_bytes": 0,
          "index_writer_memory_in_bytes": 0,
          "version_map_memory_in_bytes": 0,
          "fixed_bit_set_memory_in_bytes": 0,
          "max_unsafe_auto_id_timestamp": -9223372036854776000,
          "remote_store": {
            "upload": {
              "total_uploads": {
                "started_bytes": 0,
                "succeeded_bytes": 0,
                "failed_bytes": 0
              },
              "max_refresh_time_lag_in_millis": 0,
              "max_refresh_size_lag_in_bytes": 0
            },
            "download": {
              "total_downloads": {
                "started_bytes": 56529,
                "succeeded_bytes": 56529,
                "failed_bytes": 0
              }
            }
          },
          "file_sizes": {}
        }
      }
    },
    "EDUazBdFTYK9-Wij2HtTiQ": {
      "timestamp": 1691500774602,
      "name": "opensearch-node1",
      "transport_address": "172.18.0.2:9300",
      "host": "172.18.0.2",
      "ip": "172.18.0.2:9300",
      "roles": [
        "data",
        "ingest",
        "remote_cluster_client"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "indices": {
        "segments": {
          "count": 3,
          "memory_in_bytes": 0,
          "terms_memory_in_bytes": 0,
          "stored_fields_memory_in_bytes": 0,
          "term_vectors_memory_in_bytes": 0,
          "norms_memory_in_bytes": 0,
          "points_memory_in_bytes": 0,
          "doc_values_memory_in_bytes": 0,
          "index_writer_memory_in_bytes": 205452,
          "version_map_memory_in_bytes": 426,
          "fixed_bit_set_memory_in_bytes": 0,
          "max_unsafe_auto_id_timestamp": -1,
          "remote_store": {
            "upload": {
              "total_uploads": {
                "started_bytes": 56759,
                "succeeded_bytes": 56759,
                "failed_bytes": 0
              },
              "max_refresh_time_lag_in_millis": 0,
              "max_refresh_size_lag_in_bytes": 0
            },
            "download": {
              "total_downloads": {
                "started_bytes": 0,
                "succeeded_bytes": 0,
                "failed_bytes": 0
              }
            }
          },
          "file_sizes": {}
        }
      }
    }
  }
}

Added integ test cases to ensure that:

These stats are showing up as 0 for nodes without any remote-store backed indices
These stats also shows up as 0 for cluster manager nodes

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Shourya Dutta Biswas <[email protected]>

github-actions · 2023-08-08T13:43:56Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22099/
CommitID: 3ac12e9
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-08T14:04:42Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 14s

Signed-off-by: Shourya Dutta Biswas <[email protected]>

opensearch-trigger-bot · 2023-08-08T18:20:31Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 32s

github-actions · 2023-08-08T18:22:22Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22135/
CommitID: ce2bb70
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <[email protected]>

opensearch-trigger-bot · 2023-08-08T19:00:51Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/security-analytics.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 29m 42s

github-actions · 2023-08-08T19:13:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22141/
CommitID: 33b4f86
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <[email protected]>

github-actions · 2023-08-09T05:52:20Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22194/
CommitID: 6ce0cc2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-09T05:59:56Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 33m

opensearch-trigger-bot · 2023-08-09T06:19:38Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 27m 27s

github-actions · 2023-08-09T06:24:39Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22199/
CommitID: ee1b224
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <[email protected]>

github-actions · 2023-08-09T06:37:31Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22203/
CommitID: 439ab16
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <[email protected]>

opensearch-trigger-bot · 2023-08-09T06:57:55Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/reporting.git]

BUILD SUCCESSFUL in 27m 47s

opensearch-trigger-bot · 2023-08-09T07:12:32Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 31m 54s

github-actions · 2023-08-09T07:13:04Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22204/
CommitID: 39f8111
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: Shourya Dutta Biswas <[email protected]>

opensearch-trigger-bot · 2023-08-09T10:08:40Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 28m 18s

github-actions · 2023-08-09T10:21:25Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/22218/
CommitID: 723d4b1

codecov · 2023-08-09T10:22:20Z

Codecov Report

Merging #9168 (ef0f6e2) into main (1162865) will increase coverage by 0.04%.
Report is 16 commits behind head on main.
The diff coverage is 93.85%.

@@             Coverage Diff              @@
##               main    #9168      +/-   ##
============================================
+ Coverage     71.03%   71.07%   +0.04%     
- Complexity    57318    57404      +86     
============================================
  Files          4770     4771       +1     
  Lines        270540   270623      +83     
  Branches      39561    39561              
============================================
+ Hits         192186   192356     +170     
+ Misses        62147    62081      -66     
+ Partials      16207    16186      -21

Files Changed	Coverage Δ
...rc/main/java/org/opensearch/index/store/Store.java	`80.25% <ø> (+0.96%)`	⬆️
...ch/indices/recovery/PeerRecoveryTargetService.java	`52.30% <0.00%> (-5.27%)`	⬇️
...ava/org/opensearch/index/engine/SegmentsStats.java	`72.44% <72.72%> (-1.70%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`69.79% <88.88%> (+0.71%)`	⬆️
...rg/opensearch/index/engine/ReplicaFileTracker.java	`91.66% <91.66%> (ø)`
...rg/opensearch/index/remote/RemoteSegmentStats.java	`97.61% <97.61%> (ø)`
.../opensearch/index/engine/NRTReplicationEngine.java	`74.37% <100.00%> (+0.48%)`	⬆️
...earch/index/store/RemoteSegmentStoreDirectory.java	`88.85% <100.00%> (+0.98%)`	⬆️
...ices/replication/RemoteStoreReplicationSource.java	`90.90% <100.00%> (+0.16%)`	⬆️
.../indices/replication/SegmentReplicationTarget.java	`89.00% <100.00%> (-0.72%)`	⬇️

... and 469 files with indirect coverage changes

Signed-off-by: Shourya Dutta Biswas <[email protected]>

github-actions · 2023-08-10T05:34:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22312/
CommitID: 1e76f0b
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/index/engine/SegmentsStats.java

server/src/main/java/org/opensearch/index/remote/RemoteSegmentStats.java

server/src/internalClusterTest/java/org/opensearch/indices/stats/IndexStatsIT.java

.../internalClusterTest/java/org/opensearch/remotestore/RemoteSegmentStatsFromNodesStatsIT.java

Signed-off-by: Shourya Dutta Biswas <[email protected]>

github-actions · 2023-08-10T19:05:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22337/
CommitID: 043666a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-10T19:12:03Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22338/
CommitID: 9654b55
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-08-11T05:59:14Z

Compatibility status:



> Task :checkCompatibility
Incompatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/performance-analyzer.git]
Compatible components: [https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git]

BUILD SUCCESSFUL in 31m 18s

github-actions · 2023-08-11T06:12:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/22433/
CommitID: ef0f6e2
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

sachinpkale · 2023-08-14T03:29:59Z

Overall changes LGTM.

sachinpkale · 2023-08-14T03:31:20Z

Build is failing due to flaky test: #9116
Re-triggered.

github-actions · 2023-08-14T04:14:32Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobContainerRetriesTests.testWriteBlobWithRetries

URL: https://build.ci.opensearch.org/job/gradle-check/22598/
CommitID: ef0f6e2
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <[email protected]> (cherry picked from commit c9dbd90)

ashking94 · 2023-08-16T04:16:24Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -1377,6 +1384,12 @@ public MergeStats mergeStats() {
    public SegmentsStats segmentStats(boolean includeSegmentFileSizes, boolean includeUnloadedSegments) {
        SegmentsStats segmentsStats = getEngine().segmentsStats(includeSegmentFileSizes, includeUnloadedSegments);
        segmentsStats.addBitsetMemoryInBytes(shardBitsetFilterCache.getMemorySizeInBytes());
+        // Populate remote_store stats only if the index is remote store backed


We are doing an add here and one explicit add in NoOpEngine.java class -

OpenSearch/server/src/main/java/org/opensearch/index/engine/NoOpEngine.java

Line 140 in 4871e25

stats.add(this.segmentsStats);

This seem to be adding same stats twice.

ashking94 · 2023-08-16T04:18:37Z

server/src/main/java/org/opensearch/index/remote/RemoteSegmentStats.java

+        downloadBytesFailed = in.readLong();
+        downloadBytesSucceeded = in.readLong();
+        maxRefreshTimeLag = in.readLong();
+        maxRefreshBytesLag = in.readLong();


can this be cumulative refresh bytes lag itself?

ashking94

left some comments. do take a look.

…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <[email protected]> (cherry picked from commit c9dbd90)

…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <[email protected]> Signed-off-by: Kaushal Kumar <[email protected]>

…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <[email protected]> Signed-off-by: Ivan Brusic <[email protected]>

…pensearch-project#9168) --------- Signed-off-by: Shourya Dutta Biswas <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

shourya035 added 4 commits August 8, 2023 14:54

Adding remote segment stats to nodesStats API

5fef8f1

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Merge branch 'opensearch-project:main' into stats-rollup

bbcbcb1

Spotless formatting

2663bf8

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Changing bytesLag to BytesValue instead of TimeValue

3ac12e9

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Add more UTs and fixed spotless checks

ce2bb70

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Add else block on remoteSegmentStats serialization

33b4f86

Signed-off-by: Shourya Dutta Biswas <[email protected]>

shourya035 added 2 commits August 9, 2023 10:56

Merge branch 'opensearch-project:main' into stats-rollup

6ce0cc2

Add Nullable to optinal remote segment stats field

ee1b224

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Changing version to v3.0.0

439ab16

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Retrigger integs

39f8111

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Add null check on XContent parser for remote stats

723d4b1

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Add javadocs

6c19313

Signed-off-by: Shourya Dutta Biswas <[email protected]>

shourya035 changed the title ~~[Remote Store] Remote segment transfer stats on NodesStats API~~ [Remote Store] Add remote segment transfer stats on NodesStats API Aug 9, 2023

shourya035 requested review from sachinpkale and sohami as code owners August 9, 2023 13:24

Merge branch 'main' into stats-rollup

1e76f0b

Signed-off-by: Shourya Dutta Biswas <[email protected]>

sachinpkale reviewed Aug 10, 2023

View reviewed changes

shourya035 added 2 commits August 10, 2023 23:54

Addressing comments

043666a

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Merge branch 'main' into stats-rollup

9654b55

Signed-off-by: Shourya Dutta Biswas <[email protected]>

Merge branch 'opensearch-project:main' into stats-rollup

ef0f6e2

sachinpkale approved these changes Aug 14, 2023

View reviewed changes

sachinpkale merged commit c9dbd90 into opensearch-project:main Aug 14, 2023

shourya035 deleted the stats-rollup branch August 14, 2023 10:55

shourya035 mentioned this pull request Aug 14, 2023

[Backport-2.x] Add remote segment transfer stats on NodesStats API #9301

Merged

6 tasks

ashking94 reviewed Aug 16, 2023

View reviewed changes

shourya035 mentioned this pull request Aug 17, 2023

[Remote Store] Add cumulative bytes lag to NodesStats #9393

Merged

6 tasks

ashking94 mentioned this pull request Aug 28, 2023

[DOC] Update node stats api documentation for remote store related fields in indices segments section opensearch-project/documentation-website#4902

Closed

4 tasks

sachinpkale mentioned this pull request Sep 6, 2023

[Meta] Remote Store: 2.10.0 - Release Tracking #9805

Closed

31 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add remote segment transfer stats on NodesStats API #9168

[Remote Store] Add remote segment transfer stats on NodesStats API #9168

shourya035 commented Aug 8, 2023 •

edited

Loading

github-actions bot commented Aug 8, 2023

opensearch-trigger-bot bot commented Aug 8, 2023

opensearch-trigger-bot bot commented Aug 8, 2023

github-actions bot commented Aug 8, 2023

opensearch-trigger-bot bot commented Aug 8, 2023

github-actions bot commented Aug 8, 2023

github-actions bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

codecov bot commented Aug 9, 2023 •

edited

Loading

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

sachinpkale commented Aug 14, 2023

sachinpkale commented Aug 14, 2023

github-actions bot commented Aug 14, 2023

ashking94 Aug 16, 2023

ashking94 Aug 16, 2023

ashking94 left a comment

[Remote Store] Add remote segment transfer stats on NodesStats API #9168

[Remote Store] Add remote segment transfer stats on NodesStats API #9168

Conversation

shourya035 commented Aug 8, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 8, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 8, 2023

opensearch-trigger-bot bot commented Aug 8, 2023

github-actions bot commented Aug 8, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 8, 2023

github-actions bot commented Aug 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 9, 2023

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 9, 2023

github-actions bot commented Aug 9, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Aug 9, 2023 • edited Loading

Codecov Report

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Aug 11, 2023

github-actions bot commented Aug 11, 2023

Gradle Check (Jenkins) Run Completed with:

sachinpkale commented Aug 14, 2023

sachinpkale commented Aug 14, 2023

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

ashking94 Aug 16, 2023

Choose a reason for hiding this comment

ashking94 Aug 16, 2023

Choose a reason for hiding this comment

ashking94 left a comment

Choose a reason for hiding this comment

shourya035 commented Aug 8, 2023 •

edited

Loading

codecov bot commented Aug 9, 2023 •

edited

Loading