[Remote Store] Add support to reload repository metadata inplace #9569

BhumikaSaini-Amazon · 2023-08-28T05:18:36Z

Description

This PR adds the support to reload repository metadata in-place. Starting out, we limit this feature to only system repository reloads and the specific S3 repository implementation. This can be extended to make all repositories reloadable in-place in the future.

Related Issues

Resolves #9118

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-28T05:22:52Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/23577/
CommitID: 62f5929
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-28T05:23:19Z

Compatibility status:

Checks if related components are compatible with change 45721c1

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git]

server/src/main/java/org/opensearch/repositories/RepositoriesService.java

plugins/repository-s3/src/main/java/org/opensearch/repositories/s3/S3BlobStore.java

server/src/main/java/org/opensearch/repositories/fs/FsRepository.java

github-actions · 2023-09-12T11:17:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25377/
CommitID: d30113f
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-19T08:17:58Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25841/
CommitID: 2f0ba74
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/repositories/Repository.java

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java

github-actions · 2023-09-19T11:38:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25844/
CommitID: ced330e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-19T12:01:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25845/
CommitID: 802b2cc
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-19T12:13:36Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25847/
CommitID: 33eee39
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-19T13:07:57Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/25848/
CommitID: 8216a25
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-09-28T05:02:13Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26430/
CommitID: c605c62
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/repositories/fs/ReloadableFsRepository.java

github-actions · 2023-10-04T16:46:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26924/
CommitID: 6eb1b37
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T17:30:15Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26934/
CommitID: 6055c09
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

BhumikaSaini-Amazon · 2023-10-04T17:32:18Z

org.opensearch.index.translog.RemoteFsTranslogTests.testMetadataFileDeletion => reported flaky in #9605

Signed-off-by: Bhumika Saini <[email protected]>

…or S3 Signed-off-by: Bhumika Saini <[email protected]>

Signed-off-by: Bhumika Saini <[email protected]>

github-actions · 2023-10-04T17:43:17Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26939/
CommitID: 45721c1
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T18:38:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26940/
CommitID: b16af58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T19:38:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26949/
CommitID: b16af58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T22:10:24Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/26987/
CommitID: b16af58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

kotwanikunal · 2023-10-04T22:13:49Z

#10372

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌

URL: https://build.ci.opensearch.org/job/gradle-check/26987/

CommitID: b16af58
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-04T22:58:14Z

Gradle Check (Jenkins) Run Completed with:

RESULT: SUCCESS ✅
URL: https://build.ci.opensearch.org/job/gradle-check/26999/
CommitID: b16af58

opensearch-trigger-bot · 2023-10-05T01:23:12Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-9569-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 179c4ef94d0176de5161a0e620bca9609dd2a051
# Push it to GitHub
git push --set-upstream origin backport/backport-9569-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-9569-to-2.x.

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]>

…ta inplace (#9569) (#10390) * [Remote Store] Add support to reload repository metadata inplace (#9569) --------- Signed-off-by: Bhumika Saini <[email protected]> * Empty commit to retrigger build Signed-off-by: Bhumika Saini <[email protected]> * Empty commit to retrigger build Signed-off-by: Bhumika Saini <[email protected]> --------- Signed-off-by: Bhumika Saini <[email protected]> Signed-off-by: Gaurav Bafna <[email protected]> Co-authored-by: Gaurav Bafna <[email protected]>

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]>

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

gbbafna reviewed Aug 28, 2023

View reviewed changes

BhumikaSaini-Amazon force-pushed the repo-md-reload branch from 62f5929 to d30113f Compare September 12, 2023 10:56

github-actions bot added bug Something isn't working Storage:Durability Issues and PRs related to the durability framework Storage Issues and PRs relating to data and metadata storage v2.10.0 labels Sep 19, 2023

BhumikaSaini-Amazon force-pushed the repo-md-reload branch 2 times, most recently from ced330e to 802b2cc Compare September 19, 2023 11:10

Bukhtawar reviewed Sep 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/Repository.java Outdated Show resolved Hide resolved

Bukhtawar reviewed Sep 19, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/blobstore/BlobStoreRepository.java Outdated Show resolved Hide resolved

BhumikaSaini-Amazon force-pushed the repo-md-reload branch from 33eee39 to 8216a25 Compare September 19, 2023 12:16

github-actions bot added the v2.11.0 Issues and PRs related to version 2.11.0 label Sep 28, 2023

BhumikaSaini-Amazon force-pushed the repo-md-reload branch from c605c62 to 8a9ceff Compare September 28, 2023 05:00

BhumikaSaini-Amazon changed the title ~~[Remote Store] Reload repository metadata inplace~~ [Remote Store] Add support to reload repository metadata inplace Sep 28, 2023

BhumikaSaini-Amazon marked this pull request as ready for review September 28, 2023 05:06

BhumikaSaini-Amazon requested review from reta, anasalkouz, andrross, CEHENKLE, dblock and setiah as code owners September 28, 2023 05:06

Bukhtawar reviewed Oct 4, 2023

View reviewed changes

server/src/main/java/org/opensearch/repositories/fs/ReloadableFsRepository.java Outdated Show resolved Hide resolved

Bhumika Saini added 5 commits October 4, 2023 23:04

Add support to reload repository metadata inplace

20f7b08

Signed-off-by: Bhumika Saini <[email protected]>

Apply spotless fixes, Pass S3BlobStoreRepositoryTests

ef2c0a0

Signed-off-by: Bhumika Saini <[email protected]>

Pass testRestrictedSettingsDefault

396ff61

Signed-off-by: Bhumika Saini <[email protected]>

Update S3RepositoryThirdPartyTests

d399e73

Signed-off-by: Bhumika Saini <[email protected]>

Add default values for STORAGE_CLASS_SETTING and CANNED_ACL_SETTING f…

d4fd332

…or S3 Signed-off-by: Bhumika Saini <[email protected]>

BhumikaSaini-Amazon force-pushed the repo-md-reload branch from 6055c09 to 45721c1 Compare October 4, 2023 17:35

Bhumika Saini added 2 commits October 4, 2023 23:07

Add reloadable check, Update metadata validation

02e8d6e

Signed-off-by: Bhumika Saini <[email protected]>

Resolve CHANGELOG conflict

b16af58

Signed-off-by: Bhumika Saini <[email protected]>

BhumikaSaini-Amazon force-pushed the repo-md-reload branch from 45721c1 to b16af58 Compare October 4, 2023 17:38

gbbafna added the backport 2.x Backport to 2.x branch label Oct 4, 2023

sachinpkale merged commit 179c4ef into opensearch-project:main Oct 5, 2023
15 of 16 checks passed

opensearch-trigger-bot bot added the backport-failed label Oct 5, 2023

BhumikaSaini-Amazon pushed a commit to BhumikaSaini-Amazon/OpenSearch that referenced this pull request Oct 5, 2023

[Remote Store] Add support to reload repository metadata inplace (ope…

40bf534

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]>

BhumikaSaini-Amazon mentioned this pull request Oct 5, 2023

[Backport 2.x] [Remote Store] Add support to reload repository metadata inplace (#9569) #10390

Merged

7 tasks

deshsidd pushed a commit to deshsidd/OpenSearch that referenced this pull request Oct 9, 2023

[Remote Store] Add support to reload repository metadata inplace (ope…

d1d3027

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]>

vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023

[Remote Store] Add support to reload repository metadata inplace (ope…

11317d2

…nsearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]>

gbbafna mentioned this pull request Apr 4, 2024

[BUG]: Non-Dynamic S3 Configuration in Single-Node Opensearch Setup Fails to Restart in GitLab CI/CD #11302

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remote Store] Add support to reload repository metadata inplace #9569

[Remote Store] Add support to reload repository metadata inplace #9569

BhumikaSaini-Amazon commented Aug 28, 2023 •

edited

Loading

github-actions bot commented Aug 28, 2023

github-actions bot commented Aug 28, 2023 •

edited

Loading

github-actions bot commented Sep 12, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 19, 2023

github-actions bot commented Sep 28, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

BhumikaSaini-Amazon commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

kotwanikunal commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

opensearch-trigger-bot bot commented Oct 5, 2023

[Remote Store] Add support to reload repository metadata inplace #9569

[Remote Store] Add support to reload repository metadata inplace #9569

Conversation

BhumikaSaini-Amazon commented Aug 28, 2023 • edited Loading

Description

Related Issues

Check List

github-actions bot commented Aug 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 28, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Sep 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Sep 28, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

BhumikaSaini-Amazon commented Oct 4, 2023

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

kotwanikunal commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Oct 4, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Oct 5, 2023

BhumikaSaini-Amazon commented Aug 28, 2023 •

edited

Loading

github-actions bot commented Aug 28, 2023 •

edited

Loading