Remote state publication

Signed-off-by: Sooraj Sinha <[email protected]>

_tuning-your-cluster/availability-and-recovery/remote-store/remote-cluster-state.md

@@ -54,12 +54,44 @@ In addition to the mandatory static settings, you can configure the following dy
 
 Setting | Default | Description
 :--- | :--- | :---
-`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | The amount of time to wait for index metadata upload to complete. Note that index metadata for separate indexes is uploaded in parallel.
-`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for global metadata upload to complete. Global metadata contains globally applicable metadata, such as templates, cluster settings, data stream metadata, and repository metadata.
-`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files. 
+`cluster.remote_store.state.index_metadata.upload_timeout` | 20s | Deprecated. Use `cluster.remote_store.state.global_metadata.upload_timeout` instead.
+`cluster.remote_store.state.global_metadata.upload_timeout` | 20s | The amount of time to wait for cluster state upload to complete.
+`cluster.remote_store.state.metadata_manifest.upload_timeout` | 20s | The amount of time to wait for the manifest file upload to complete. The manifest file contains the details of each of the files uploaded for a single cluster state, both index metadata files and global metadata files.
+`cluster.remote_store.state.cleanup_interval` | 300s | The interval for remote state clean-up async task to run. This task deletes the old remote state files. 
 
 
 ## Limitations
 
 The remote cluster state functionality has the following limitations:
 - Unsafe bootstrap scripts cannot be run when the remote cluster state is enabled. When a majority of cluster-manager nodes are lost and the cluster goes down, the user needs to replace any remaining cluster manager nodes and reseed the nodes in order to bootstrap a new cluster.
+
+## Remote Cluster State Publication
+The cluster state published to remote-backed storage can be used for publication. Currently, the active cluster manager
+sends the cluster state object over the transport layer to the follower nodes. This flow can be changed to fetch the
+cluster state from remote store. This can be done by enabling the experimental remote publication feature. 
+Enable the feature flag for `remote_store.publication` feature by following the [experiment feature flag documentation]({{site.url}}{{site.baseurl}}/install-and-configure/configuring-opensearch/experimental/).
+When remote publication is enabled, the cluster manager node uploads the cluster state to remote store and then sends the 
+remote path of the cluster state to the follower nodes. The follower nodes then download the cluster state from remote store.
+
+The routing table is an object within the cluster state which contains the shard allocation details for each index.
+This object can become large in case of large number of shards in the cluster. Routing table is required to be stored in
+remote store for the remote publication to work. In order to enable remote persistence of routing table, the repository must
+be configured as below:
+
+```yml
+# Remote routing table repository settings
+node.attr.remote_store.routing_table.repository: my-remote-routing-table-repo
+node.attr.remote_store.repository.my-remote-routing-table-repo.type: s3
+node.attr.remote_store.repository.my-remote-routing-table-repo.settings.bucket: <Bucket Name 3>
+node.attr.remote_store.repository.my-remote-routing-table-repo.settings.region: <Bucket region>
+```
+You do not have to use different remote store repositories for state and routing. 
+These stores can share the same repository.
+
+The relevant cluster settings for remote publication are listed below:
+
+Setting | Default | Description
+:--- | :--- | :---
+`cluster.remote_store.state.read_timeout` | 20s | The amount of time to wait for remote state download to complete on the follower node.
+`cluster.remote_store.routing_table.path_type` | HASHED_PREFIX | Path type to be used for creating index routing path in blob store. Valid values are "FIXED", "HASHED_PREFIX", "HASHED_INFIX"
+`cluster.remote_store.routing_table.path_hash_algo` | FNV_1A_BASE64 | Algorithm to be used for constructing prefix or infix of blob store path. This setting comes into effect into if cluster.remote_store.routing_table.path_type is "hashed_prefix" or "hashed_infix". Valid values of algo are "FNV_1A_BASE64" or "FNV_1A_COMPOSITE_1"