-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Split the remote global metadata file to metadata attribute files #12190
Split the remote global metadata file to metadata attribute files #12190
Conversation
…bute files Signed-off-by: Shivansh Arora <[email protected]>
Compatibility status:Checks if related components are compatible with change 8244c6d Incompatible componentsSkipped componentsCompatible components |
❌ Gradle check result for 6bf7bc9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for d0875f9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for f6a2431: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Shivansh Arora <[email protected]>
f3e853b
to
5d6a0ad
Compare
❌ Gradle check result for f3e853b: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 5d6a0ad: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
server/src/main/java/org/opensearch/cluster/metadata/RepositoriesMetadata.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/cluster/metadata/TemplatesMetadata.java
Show resolved
Hide resolved
server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java
Show resolved
Hide resolved
Looks good on a high level. Can you move it out of draft ? |
Signed-off-by: Shivansh Arora <[email protected]>
server/src/main/java/org/opensearch/gateway/remote/ClusterMetadataManifest.java
Outdated
Show resolved
Hide resolved
server/src/main/java/org/opensearch/gateway/remote/RemoteClusterStateService.java
Show resolved
Hide resolved
❌ Gradle check result for 279dbbe: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
❌ Gradle check result for 52d5f81: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
52d5f81
to
4f8a64e
Compare
❌ Gradle check result for 4f8a64e: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Approving this change. Lets refactor RemoteClusterStateService.java before making any further changes to this class.
…2190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92)
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92)
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]>
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92)
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92)
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92)
…ibute files (#13703) * Split the remote global metadata file to metadata attribute files (#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92) * Remove conflicting static method from Metadata.Custom interface Signed-off-by: Shivansh Arora <[email protected]> --------- Signed-off-by: Shivansh Arora <[email protected]>
…ensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]>
…ibute files (opensearch-project#13703) * Split the remote global metadata file to metadata attribute files (opensearch-project#12190) * Split the cluster state remote global metadata file to metadata attribute files Signed-off-by: Shivansh Arora <[email protected]> (cherry picked from commit da3ab92) * Remove conflicting static method from Metadata.Custom interface Signed-off-by: Shivansh Arora <[email protected]> --------- Signed-off-by: Shivansh Arora <[email protected]> Signed-off-by: kkewwei <[email protected]>
Description
We are now uploading the global metadata of a cluster state as a separate file for each metadata attribute like coordination metadata, settings, templates and all of the custom metadata attributes. Remote global state directory will look like below:
Splitting the global-metadata into multiple files have improved the incremental metadata upload time to S3 by 50-70%, and full metadata upload by upto 5% because of parallel upload of global metadata attribute and index metadata files.
These benchmarks were done by writing a microbenchmark on main (shiv0408/OpenSearch@fe5fad8) and on top of PR branch (shiv0408/OpenSearch@88ab1ac)
Following are the benchmark results:
Benchmark on main
Benchmark after splitting the global metadata
Related Issues
Resolves #12468
Resolves #10645
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.