Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add segment replication + remote store GA information #5037

Merged
merged 11 commits into from
Sep 19, 2023
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ redirect_from:

# Remote-backed storage

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
{: .warning}
Introduced 2.10
{: .label .label-purple }


Remote-backed storage offers OpenSearch users a new way to protect against data loss by automatically creating backups of all index transactions and sending them to remote storage. In order to expose this feature, segment replication must also be enabled. See [Segment replication]({{site.url}}{{site.baseurl}}/opensearch/segment-replication/) for additional information.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ grand_parent: Availability and recovery

# Remote segment backpressure

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
{: .warning}
Introduced 2.10
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
{: .label .label-purple }

Remote segment backpressure is a shard-level rejection mechanism that dynamically rejects indexing requests when the remote segment store falls behind the local committed segments on the primary shard. With remote segment backpressure, you can prevent the lag between the remote store and the local primary store. The lag can be caused by slow or failed remote store interaction, remote store throttling, long garbage collection pauses, or high CPU utilization.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,6 @@ grand_parent: Availability and recovery
Introduced 2.8
{: .label .label-purple }

This is an experimental feature and is not recommended for use in a production environment. For updates on the progress of the feature or if you want to leave feedback, see the associated [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/1968).
{: .warning}

Use the Remote Store Stats API to monitor shard-level remote store performance.

## Path and HTTP methods
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,13 @@
- When experiencing very high loads, you want to add new nodes but don't want to index all data immediately.
- OpenSearch cluster deployments with low replica counts, such as those used for log analytics.

## Remote store

As of OpenSearch 2.10, you can use two methods for segment replication:

- Using with a **Remote Store**, a persistent storage solution where you can store data more durably, the primary shard mirrors segments to the remote store and the replica shard hydrates the copy from the same store. For more information about using a remote store, see [Remote backend storage]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#segment-replication-and-remote-backed-storage)
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
- Without a remote store, where replica shards will instead sync segments from the primary shard using node-to-node communication.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Segment replication configuration

Setting the default replication type for a cluster affects all newly created indexes. You can, however, specify a different replication type when creating an index. Index-level settings override cluster-level settings.
Expand All @@ -48,7 +55,9 @@
```
{% include copy-curl.html %}

In segment replication, the primary shard is usually generating more network traffic than the replicas because it copies segment files to the replicas. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/).
If you're using a remote store, add the `remote_store` property to the index request body. For more information, see [Create an index]({{site.url}}{{site.baseurl}}/tuning-your-cluster/availability-and-recovery/remote-store/index/#create-an-index).

When using node-to-node replication, the primary shard consumes higher network bandwidth because it pushes segment files to all the replica shards. Thus, it's beneficial to distribute primary shards equally between the nodes. To ensure balanced primary shard distribution, set the dynamic `cluster.routing.allocation.balance.prefer_primary` setting to `true`. For more information, see [Cluster settings]({{site.url}}{{site.baseurl}}/api-reference/cluster-api/cluster-settings/).
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

For the best performance, it is recommended that you enable the following settings:

Expand All @@ -75,8 +84,7 @@
```
{% include copy.html %}

This cluster-level setting cannot be enabled through the [REST API]({{site.url}}{{site.baseurl}}/api-reference/index/). This setting is not applied to system indexes and hidden indexes. By default, all system and hidden indexes in OpenSearch use document replication, even if this setting is enabled.
{: .note}


### Creating an index with document replication

Expand All @@ -101,11 +109,11 @@
1. Enabling segment replication for an existing index requires [reindexing](https://github.com/opensearch-project/OpenSearch/issues/3685).
1. [Cross-cluster replication](https://github.com/opensearch-project/OpenSearch/issues/4090) does not currently use segment replication to copy between clusters.
1. Segment replication is not compatible with [document-level monitors]({{site.url}}{{site.baseurl}}/observing-your-data/alerting/api/#document-level-monitors), which are used with the [Alerting]({{site.url}}{{site.baseurl}}/install-and-configure/plugins/) and [Security Analytics]({{site.url}}{{site.baseurl}}/security-analytics/index/) plugins. The plugins also use the latest available data on replica shards when using the `immediate` refresh policy, and segment replication can delay the policy's availability, resulting in stale replica shards.
1. Segment replication leads to increased network congestion on primary shards. See [Issue - Optimize network bandwidth on primary shards](https://github.com/opensearch-project/OpenSearch/issues/4245).
1. Integration with remote-backed storage as the source of replication is [currently not supported](https://github.com/opensearch-project/OpenSearch/issues/4448).
1. Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. We are exploring other mechanisms for providing read-after-write guarantees. For more information, see the corresponding [GitHub issue](https://github.com/opensearch-project/OpenSearch/issues/6046).
1. System indexes will continue to use document replication internally until read-after-write guarantees are available. In this case, document replication does not hinder the overall performance because there are few system indexes.
1. Segment replication leads to increased network congestion on primary shards using node-to-node replication. With a remote store, the primary shard can upload segments to the remote store, then you can download replicas from the same store.
Read-after-write guarantees: Segment replication does not currently support setting the refresh policy to `wait_for`. If you set the `refresh` query parameter to `wait_for` and then ingest documents, you'll get a response only after the primary node has refreshed and made those documents searchable. Replica shards will respond only after having written to their local translog. If realtime reads are needed, consider using [`get`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/get-documents/) or [`mget`]({{site.url}}{{site.baseurl}}/api-reference/document-apis/multi-get/) API operations.

Check failure on line 113 in _tuning-your-cluster/availability-and-recovery/segment-replication/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _tuning-your-cluster/availability-and-recovery/segment-replication/index.md#L113

[OpenSearch.Spelling] Error: realtime. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output
{"message": "[OpenSearch.Spelling] Error: realtime. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_tuning-your-cluster/availability-and-recovery/segment-replication/index.md", "range": {"start": {"line": 113, "column": 385}}}, "severity": "ERROR"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
1. As of OpenSearch 2.10, system indexes are now supported inside segment replication.
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved
1. Get, MultiGet, TermVector, and MultiTermVector requests serve strong reads by routing requests to the primary shards. This can hurt performance since more requests are handled by the primary shards versus distributing requests across primary and replicate shards. To help with performance in read heavy clusters, we recommend setting the `realtime` parameter in these requests to `false`, especially with listed request types. For more information, see [Issue #8700](https://github.com/opensearch-project/OpenSearch/issues/8700).

Check warning on line 115 in _tuning-your-cluster/availability-and-recovery/segment-replication/index.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _tuning-your-cluster/availability-and-recovery/segment-replication/index.md#L115

[OpenSearch.LatinismsSubstitution] Use 'compared to or compared with' instead of 'versus'.
Raw output
{"message": "[OpenSearch.LatinismsSubstitution] Use 'compared to or compared with' instead of 'versus'.", "location": {"path": "_tuning-your-cluster/availability-and-recovery/segment-replication/index.md", "range": {"start": {"line": 115, "column": 202}}}, "severity": "WARNING"}
Naarcha-AWS marked this conversation as resolved.
Show resolved Hide resolved

## Benchmarks

During initial benchmarks, segment replication users reported 40% higher throughput than when using document replication with the same cluster setup.
Expand Down