diff --git a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md index 7cc533fe76..8b7dacffc8 100644 --- a/_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md +++ b/_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md @@ -54,12 +54,44 @@ In addition to the mandatory static settings, you can configure the following dy Setting | Default | Description :--- | :--- | :--- -`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | The amount of time to wait for index metadata upload to complete. Note that index metadata for separate indexes is uploaded in parallel. -`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for global metadata upload to complete. Global metadata contains globally applicable metadata, such as templates, cluster settings, data stream metadata, and repository metadata. -`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files. +`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | Deprecated. Use `cluster.remote_store.state.global_metadata.upload_timeout` instead. +`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for cluster state upload to complete. +`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files. +`cluster.remote_store.state.cleanup_interval` | 300s | The interval for remote state clean-up async task to run. This task deletes the old remote state files. ## Limitations The remote cluster state functionality has the following limitations: - Unsafe bootstrap scripts cannot be run when the remote cluster state is enabled. When a majority of cluster-manager nodes are lost and the cluster goes down, the user needs to replace any remaining cluster manager nodes and reseed the nodes in order to bootstrap a new cluster. + +## Remote Cluster State Publication +The cluster state published to remote-backed storage can be used for publication. Currently, the active cluster manager +sends the cluster state object over the transport layer to the follower nodes. This flow can be changed to fetch the +cluster state from remote store. This can be done by enabling the experimental remote publication feature. +Enable the feature flag for `remote_store.publication` feature by following the [experiment feature flag documentation]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/). +When remote publication is enabled, the cluster manager node uploads the cluster state to remote store and then sends the +remote path of the cluster state to the follower nodes. The follower nodes then download the cluster state from remote store. + +The routing table is an object within the cluster state which contains the shard allocation details for each index. +This object can become large in case of large number of shards in the cluster. Routing table is required to be stored in +remote store for the remote publication to work. In order to enable remote persistence of routing table, the repository must +be configured as below: + +```yml +# Remote routing table repository settings +node.attr.remote_store.routing_table.repository: my-remote-routing-table-repo +node.attr.remote_store.repository.my-remote-routing-table-repo.type: s3 +node.attr.remote_store.repository.my-remote-routing-table-repo.settings.bucket: +node.attr.remote_store.repository.my-remote-routing-table-repo.settings.region: +``` +You do not have to use different remote store repositories for state and routing. +These stores can share the same repository. + +The relevant cluster settings for remote publication are listed below: + +Setting | Default | Description +:--- | :--- | :--- +`cluster.remote_store.state.read_timeout` | 20s | The amount of time to wait for remote state download to complete on the follower node. +`cluster.remote_store.routing_table.path_type` | HASHED_PREFIX | Path type to be used for creating index routing path in blob store. Valid values are "FIXED", "HASHED_PREFIX", "HASHED_INFIX" +`cluster.remote_store.routing_table.path_hash_algo` | FNV_1A_BASE64 | Algorithm to be used for constructing prefix or infix of blob store path. This setting comes into effect into if cluster.remote_store.routing_table.path_type is "hashed_prefix" or "hashed_infix". Valid values of algo are "FNV_1A_BASE64" or "FNV_1A_COMPOSITE_1"