Adds support to download Remote Cluster State marker file #2

linuxpi · 2023-08-10T10:21:22Z

Description

refactor listByPrefixSortedOrder method to provide flexibility to use it without listener. Most of the usages today don't need the listener based async impl. With current refactoring, we can use it both ways -- sync and async.
Adds utilities to download latest/any Remote Cluster State marker file

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2023-08-10T10:22:07Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: a499ad0
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-10T10:24:59Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 77cac81
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-10T12:13:45Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: e8226d4
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-08-14T19:40:10Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: 1680f3e
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

Signed-off-by: bansvaru <[email protected]>

…tResetAndAbort is flaky (opensearch-project#9206) Signed-off-by: Andriy Redko <[email protected]>

Signed-off-by: Thomas Farr <[email protected]>

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <[email protected]>

…-project#9082) This is the first round refactor of the foundation Task classes from the server module to core and common library. This is split into separate commits due to control the surface area impacted by the change. This commit refactors the following classes: * o.o.action.ActionFuture -> :libs:opensearch-common * o.o.action.ActionListener -> :libs:opensearch-core * o.o.action.ActionResponse -> :libs:opensearch-core * o.o.action.NotifyOnceListener -> :libs:opensearch-core * o.o.common.compress.Compressor -> :libs:opensearch-core * o.o.node.ReportingService -> o.o.core.service in :libs:opensearch-core * o.o.tasks.TaskCancelledExceptoin -> :libs:opensearch-core * o.o.tasks.TaskId -> :libs:opensearch-core * o.o.common.xcontent.ObjectParserHelper -> :libs:opensearch-core The following task resource tracking classes are refactored into a new resourcetracker namespace under the o.o.core.tasks packed in :libs:opensearch-core * o.o.tasks.ResourceStats * o.o.tasks.ResourceStatsType * o.o.tasks.ResourceUsageMetric * o.o.tasks.TaskResourceStats * o.o.tasks.TaskResourceUsage * o.o.tasks.TaskThreadUsage * o.o.tasks.ThreadResourceInfo These refactors are part of the effort to split foundation classes into core libraries in support of modularity for serverless and cloud native implementations. Signed-off-by: Nicholas Walter Knize <[email protected]>

…nsearch-project#9221) Signed-off-by: Andriy Redko <[email protected]>

…/repository-azure (opensearch-project#9231) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) Signed-off-by: bansvaru <[email protected]> --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <[email protected]> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <[email protected]> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Add more shard level tests. Signed-off-by: Marc Handalian <[email protected]> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <[email protected]> * Self review. Signed-off-by: Marc Handalian <[email protected]> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>

…9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. Signed-off-by: bansvaru <[email protected]> --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…earch-project#9245) Signed-off-by: bansvaru <[email protected]>

…search-project#9252) * Force upgrade the vulnerable dependencies of hadoop-minicluster Signed-off-by: Thomas Farr <[email protected]> * Add changelog entry Signed-off-by: Thomas Farr <[email protected]> --------- Signed-off-by: Thomas Farr <[email protected]>

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <[email protected]>

…itory-gcs (opensearch-project#9230) * Bump com.google.code.gson:gson in /plugins/repository-gcs Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.9.0 to 2.10.1. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10.1) Signed-off-by: bansvaru <[email protected]> --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…t#9228) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.2.0 to 1.8.0. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v1.2.0...v1.8.0) Signed-off-by: bansvaru <[email protected]> --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…-project#9244) Signed-off-by: Marc Handalian <[email protected]>

…est (opensearch-project#9079) * Unmute testIndexingWithSegRep rolling upgrade test Signed-off-by: Suraj Singh <[email protected]> * Assert on row count before processing the result Signed-off-by: Suraj Singh <[email protected]> * Filter rows with 0 doc count Signed-off-by: Suraj Singh <[email protected]> * Retry assertHitCount Signed-off-by: Suraj Singh <[email protected]> * Handle exception Signed-off-by: Suraj Singh <[email protected]> * Add comment Signed-off-by: Suraj Singh <[email protected]> --------- Signed-off-by: Suraj Singh <[email protected]>

…r meaning and relations between them (opensearch-project#9223) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]>

Signed-off-by: Jay Deng <[email protected]>

…RestoreService Signed-off-by: bansvaru <[email protected]>

…toreService Signed-off-by: bansvaru <[email protected]>

Signed-off-by: bansvaru <[email protected]>

…atin checks Signed-off-by: bansvaru <[email protected]>

Signed-off-by: bansvaru <[email protected]>

github-actions · 2023-08-16T10:23:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: null ❌
URL:
CommitID: c1cb834
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

soosinha and others added 3 commits August 8, 2023 07:48

Upload all index metadata to remote

327dc4c

Merge branch 'opensearch-project:main' into remote_state

dbe33d9

Refactor changes

495ec89

linuxpi changed the base branch from main to remote_state August 10, 2023 10:21

soosinha added 3 commits August 11, 2023 14:19

Remove redundant methods

2e53375

incremental metadata upload

9806f28

Change serialize logic

2386402

linuxpi force-pushed the remote-state-marker-download branch from e8226d4 to 1680f3e Compare August 14, 2023 19:39

soosinha and others added 18 commits August 15, 2023 12:15

Update serialization logic for ClusterMetadataMarker

3873b77

Add support to download Remote Cluster State marker file

1caca7a

Signed-off-by: bansvaru <[email protected]>

Remove redudentent methods

5723f54

Signed-off-by: bansvaru <[email protected]>

Add support to download IndexMetadata from Remote Cluster State

116f3bd

Signed-off-by: bansvaru <[email protected]>

[BUG] org.opensearch.client.RestClientSingleHostIntegTests.testReques…

501343b

…tResetAndAbort is flaky (opensearch-project#9206) Signed-off-by: Andriy Redko <[email protected]>

Add release notes for 1.3.12 (opensearch-project#9214)

d626bc9

Signed-off-by: Thomas Farr <[email protected]>

[Remote Store] Fix couple of Remote Store flaky test and use bulk api…

402b817

… for ingestion (opensearch-project#9190) --------- Signed-off-by: Gaurav Bafna <[email protected]>

Automatically update GitHub Actions dependencies with dependabot (ope…

188fbd3

…nsearch-project#9221) Signed-off-by: Andriy Redko <[email protected]>

Fix test SegmentReplicationIndexShardTests.testPrimaryRestart. (opens…

efb2b88

…earch-project#9245) Signed-off-by: bansvaru <[email protected]>

[Remote Store] Introduce RemoteStoreRestoreService to handle restore …

39e31e9

…from remote store (opensearch-project#9256) Signed-off-by: bansvaru <[email protected]>

Fix flaky test SegmentReplicationIT.testDeleteOperations. (opensearch…

46cd627

…-project#9244) Signed-off-by: Marc Handalian <[email protected]>

dreamer-89 and others added 17 commits August 16, 2023 14:59

docCountError must be 0 for shard level reduce (opensearch-project#9246)

25dc45a

Signed-off-by: Jay Deng <[email protected]>

Add support to remote restore IndexMetadata and introduce RemoteStore…

2b70379

…RestoreService Signed-off-by: bansvaru <[email protected]>

Reuse RestoreCompletionResponse from RestoreService in RemoteStoreRes…

88a213c

…toreService Signed-off-by: bansvaru <[email protected]>

Minor Refactoring and Spotless fixes

1873f78

Signed-off-by: bansvaru <[email protected]>

fail restore for same index name and uuid. also harden existing valid…

226bbb1

…atin checks Signed-off-by: bansvaru <[email protected]>

add first integ test and minor fixes to logic

86ea74e

Signed-off-by: bansvaru <[email protected]>

add docstring for RemoteStoreRestoreService

2870b89

Signed-off-by: bansvaru <[email protected]>

Refactor integ test code

ca61f61

Signed-off-by: bansvaru <[email protected]>

Remove shortcut changes

9bf7636

Signed-off-by: bansvaru <[email protected]>

Final Refactor

92270a1

Signed-off-by: bansvaru <[email protected]>

Add clusterUUID param support in Remote Restore API

a9ad9bd

Signed-off-by: bansvaru <[email protected]>

rebase with latest changes on main

1359b8e

Signed-off-by: bansvaru <[email protected]>

refactor integ tests and incorporate PR comments

0c2db92

Signed-off-by: bansvaru <[email protected]>

remove stale integ test override

35a4f62

Signed-off-by: bansvaru <[email protected]>

integrate restore changes with state upload and download

c1cb834

Signed-off-by: bansvaru <[email protected]>

linuxpi force-pushed the remote-state-marker-download branch from 1680f3e to c1cb834 Compare August 16, 2023 10:22

soosinha force-pushed the remote_state branch 5 times, most recently from 50d6db3 to a4f1137 Compare August 21, 2023 08:50

soosinha force-pushed the remote_state branch 6 times, most recently from 2506421 to 4824bb5 Compare August 30, 2023 06:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support to download Remote Cluster State marker file #2

Adds support to download Remote Cluster State marker file #2

linuxpi commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 10, 2023

github-actions bot commented Aug 14, 2023

github-actions bot commented Aug 16, 2023

Adds support to download Remote Cluster State marker file #2

Are you sure you want to change the base?

Adds support to download Remote Cluster State marker file #2

Conversation

linuxpi commented Aug 10, 2023

Description

Related Issues

Check List

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 10, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 14, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Aug 16, 2023

Gradle Check (Jenkins) Run Completed with: