[BUG] IndexSettings gets into invalid states on a cluster

[peternied]

Describe the bug

During snapshot restore and other unknown scenarios settings are not validated or migrated to be compatible with future versions allowing clusters to get into states where new nodes cannot join the cluster because of validation checks. I have debugged clusters in this state, and have found records indicating these symptom has occurred many times.

Related component

Storage:Snapshots

To Reproduce

While I've seen clusters with green indices in this state, I don't know how that repo was possible

1. Create a snapshot
2. Restore snapshot with settings

POST /_snapshot/my-opensearch-repo/my-first-snapshot/_restore
{
  "indices": "opendistro-reports-definitions",
  "ignore_unavailable": true,
  "include_global_state": false,
  "index_settings": {
     "index.mapper.dynamic": true
  }
}

Alternative Repro

1. Checkout this branch main...peternied:OpenSearch-1:repro-restore-to-invalid-state
2. ./gradlew :server:internalClusterTest --tests org.opensearch.snapshots.RestoreSnapshotInvalidStateIT

Expected behavior

Indices should not be created that are in invalid states, it seems there are ways to modifed IndexSettings that bypass checks - but prevent reloading in future scenarios.

MapperService

In the specific repo index.mapper.dynamic is not check as being valid during creation, but when MapperService is constructed. While this is discoverable when looking at failed shard counts, the index still exists in the cluster state.

OpenSearch/server/src/main/java/org/opensearch/index/mapper/MapperService.java

Lines 263 to 265 in b19e427

	if (INDEX_MAPPER_DYNAMIC_SETTING.exists(indexSettings.getSettings())) {
	throw new IllegalArgumentException("Setting " + INDEX_MAPPER_DYNAMIC_SETTING.getKey() + " was removed after version 6.0.0");
	}

MergeSchedulerConfig

There was a case where the max_thread_count was lower than max_merge_count. When IndicesClusterStateService.deleteIndices(...) was called it threw an exception because it couldn't create the IndexSettings.

OpenSearch/server/src/main/java/org/opensearch/index/MergeSchedulerConfig.java

Lines 139 to 143 in b19e427

	if (maxThreadCount > maxMergeCount) {
	throw new IllegalArgumentException(
	"maxThreadCount (= " + maxThreadCount + ") should be <= maxMergeCount (= " + maxMergeCount + ")"
	);
	}

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):
OpenSearch 2.3 & OpenSearch 2.7

Additional context

• Warn about index.mapper.dynamic usage instead of erroring #11193
• [BUG]For ES version after 6.0.0 the index seems to be created with " index.mapper.dynamic " attribute which ideally it should not have #3879