Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add documentation for remote store migration #7121

Merged
merged 11 commits into from
May 13, 2024
Merged

Add documentation for remote store migration #7121

merged 11 commits into from
May 13, 2024

Conversation

gbbafna
Copy link
Contributor

@gbbafna gbbafna commented May 9, 2024

Description

Issues Resolved

resolves #7120
List any issues this PR will resolve, e.g. Closes [...].

Checklist

  • By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
    For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@gbbafna gbbafna force-pushed the remote-migration branch from 4b6cbc9 to 1852309 Compare May 9, 2024 13:53
@gbbafna gbbafna force-pushed the remote-migration branch from 1852309 to 0e13e40 Compare May 9, 2024 14:05
@gbbafna gbbafna changed the title add documentation for remote store migration Add documentation for remote store migration May 9, 2024
@Naarcha-AWS Naarcha-AWS self-assigned this May 9, 2024
@Naarcha-AWS Naarcha-AWS added release-notes PR: Include this PR in the automated release notes v2.14.0 4 - Doc review PR: Doc review in progress 3 - Tech review PR: Tech review in progress and removed 4 - Doc review PR: Doc review in progress labels May 9, 2024
shourya035
shourya035 approved these changes May 10, 2024
View reviewed changes
Copy link
Member

@shourya035 shourya035 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from the technical standpoint. Approving.

BhumikaSaini-Amazon
BhumikaSaini-Amazon suggested changes May 10, 2024
View reviewed changes
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
gbbafna and others added 3 commits May 10, 2024 17:38
@Naarcha-AWS Naarcha-AWS added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress labels May 10, 2024
Naarcha-AWS
Naarcha-AWS requested changes May 10, 2024
View reviewed changes
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Show resolved Hide resolved
@gbbafna @Naarcha-AWS
Apply suggestions from code review
a22236e 
Co-authored-by: Naarcha-AWS <[email protected]>
Signed-off-by: Gaurav Bafna <[email protected]>
BhumikaSaini-Amazon
BhumikaSaini-Amazon reviewed May 13, 2024
View reviewed changes
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated
PUT "/_cluster/settings?pretty"
{
"persistent": {
"remote_store.compatibility_mode": null,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried running this but it is failing currently (see opensearch-project/OpenSearch#13634). Maybe for 2.14 we can add a step to explicitly set this to strict? In 2.15 we can update the docs once we have the bugfix in.

@Naarcha-AWS Naarcha-AWS added 5 - Editorial review PR: Editorial review in progress and removed 4 - Doc review PR: Doc review in progress labels May 13, 2024
natebower
natebower reviewed May 13, 2024
View reviewed changes
Copy link
Collaborator

@natebower natebower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gbbafna @Naarcha-AWS Please see my comments and changes and let me know if you have any questions. Thanks!

_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated
}
```

1. Set the `remote_store.compatibility_mode` setting to `mixed` to allow remote-store backed nodes to join the cluster. Then, set `migration.direction` to `remote_store`, which makes new indexes are allocated to remote backed data nodes. The following example updates the aforementioned setting using the Cluster settings API:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First sentence: "remote-store backed" => "remote-backed storage"? Otherwise, "remote-store-backed"

_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated Show resolved Hide resolved
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated
| Field | Data type | Description |
| :--- |:----------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| remote_store.compatibility_mode | String | Defaults to`strict` mode where it only allows either non-remote or remote nodes depending upon the initial cluster type. When set to `mixed`, it allows remote and non-remote nodes to join the cluster. | |
| migration.direction | String | Defaults to `none` . `remote_store` direction creates new shards only on remote store backed nodes. |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A direction cannot technically create anything. Either "remote-backed storage nodes" or "remote-store-backed nodes".

Naarcha-AWS and others added 2 commits May 13, 2024 10:47
@Naarcha-AWS @natebower
Apply suggestions from code review
85cea85 
Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
@Naarcha-AWS
Update migrating-to-remote.md
52a7d44 
Signed-off-by: Naarcha-AWS <[email protected]>
Naarcha-AWS
Naarcha-AWS reviewed May 13, 2024
View reviewed changes
_tuning-your-cluster/availability-and-recovery/remote-store/migrating-to-remote.md Outdated
| Field | Data type | Description |
| :--- |:----------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `remote_store.compatibility_mode` | String | Defaults to `strict` mode, which only allows either non-remote or remote nodes, depending upon the initial cluster type. When set to `mixed`, it allows remote and non-remote nodes to join the cluster. | |
| `migration.direction` | String | Creates new shards only on remote-backed storage nodes. Default is `None`. |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
| `migration.direction` | String | Creates new shards only on remote-backed storage nodes. Default is `None`. |
| `migration.direction` | String | Creates new shards only on remote-backed storage nodes. Default is `None`. |

@Naarcha-AWS Naarcha-AWS merged commit debc0df into opensearch-project:main May 13, 2024
6 checks passed
@hdhalter hdhalter added 3 - Done Issue is done/complete and removed 5 - Editorial review PR: Editorial review in progress labels May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BhumikaSaini-Amazon BhumikaSaini-Amazon BhumikaSaini-Amazon requested changes

@natebower natebower natebower left review comments

@Naarcha-AWS Naarcha-AWS Naarcha-AWS approved these changes

@shourya035 shourya035 shourya035 approved these changes

@hdhalter hdhalter Awaiting requested review from hdhalter

@kolchfa-aws kolchfa-aws Awaiting requested review from kolchfa-aws kolchfa-aws is a code owner

@vagimeli vagimeli Awaiting requested review from vagimeli

@AMoo-Miki AMoo-Miki Awaiting requested review from AMoo-Miki AMoo-Miki is a code owner

@dlvenable dlvenable Awaiting requested review from dlvenable dlvenable is a code owner

@stephen-crawford stephen-crawford Awaiting requested review from stephen-crawford

@epugh epugh Awaiting requested review from epugh epugh is a code owner

Assignees

@Naarcha-AWS Naarcha-AWS

Labels
3 - Done Issue is done/complete experimental release-notes PR: Include this PR in the automated release notes v2.14.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[DOC] Add documentation for remote store migration support
6 participants
@gbbafna @Naarcha-AWS @natebower @BhumikaSaini-Amazon @shourya035 @hdhalter