[Backport 2.x] Refactor remote store flow to support any path type #12822 #12920

[ashking94]

Backporting 3907ec9 from #12822 and f6d6fd3 from #12920

[github-actions]

❕ Gradle check result for a7f90ad: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteStoreStatsIT.testDownloadStatsCorrectnessSinglePrimaryMultipleReplicaShards

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 85.89744% with 11 lines in your changes are missing coverage. Please review.

Project coverage is 71.17%. Comparing base (0dd892c) to head (a7f90ad).
Report is 189 commits behind head on 2.x.

Files	Patch %	Lines
...h/cluster/metadata/MetadataCreateIndexService.java	83.33%	0 Missing and 1 partial ⚠️
...c/main/java/org/opensearch/index/IndexService.java	0.00%	1 Missing ⚠️
.../main/java/org/opensearch/index/IndexSettings.java	75.00%	0 Missing and 1 partial ⚠️
.../opensearch/index/remote/RemoteStoreDataEnums.java	94.44%	1 Missing ⚠️
...g/opensearch/index/remote/RemoteStorePathType.java	87.50%	1 Missing ⚠️
...arch/index/remote/RemoteStorePathTypeResolver.java	80.00%	1 Missing ⚠️
...in/java/org/opensearch/index/shard/IndexShard.java	80.00%	1 Missing ⚠️
...ndex/store/RemoteSegmentStoreDirectoryFactory.java	87.50%	0 Missing and 1 partial ⚠️
...rg/opensearch/index/translog/RemoteFsTranslog.java	87.50%	0 Missing and 1 partial ⚠️
...dex/translog/transfer/TranslogTransferManager.java	87.50%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x   #13405      +/-   ##
============================================
- Coverage     71.28%   71.17%   -0.11%     
- Complexity    60145    60727     +582     
============================================
  Files          4957     5014      +57     
  Lines        282799   285902    +3103     
  Branches      41409    41790     +381     
============================================
+ Hits         201591   203505    +1914     
- Misses        64189    65223    +1034     
- Partials      17019    17174     +155     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ashking94]

The Detect Breaking Changes failure is on changes that are either unreleased or internally only used ctors/methods.

Comparing source compatibility of  against 
+++  NEW CLASS: PUBLIC(+) org.opensearch.common.blobstore.BlobPath  (not serializable)
	+++  CLASS FILE FORMAT VERSION: 55.0 <- n.a.
	+++  NEW INTERFACE: java.lang.Iterable
	+++  NEW SUPERCLASS: java.lang.Object
	+++  NEW CONSTRUCTOR: PUBLIC(+) BlobPath()
	+++  NEW METHOD: PUBLIC(+) org.opensearch.common.blobstore.BlobPath add(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) java.lang.String buildAsString()
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.common.blobstore.BlobPath cleanPath()
	+++  NEW METHOD: PUBLIC(+) java.util.Iterator<java.lang.String> iterator()
	+++  NEW METHOD: PUBLIC(+) org.opensearch.common.blobstore.BlobPath parent()
		+++  NEW ANNOTATION: org.opensearch.common.Nullable
	+++  NEW METHOD: PUBLIC(+) java.lang.String[] toArray()
	+++  NEW METHOD: PUBLIC(+) java.lang.String toString()
***  MODIFIED CLASS: PUBLIC FINAL org.opensearch.index.IndexSettings  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathType getRemoteStorePathType()
+++  NEW ENUM: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory  (compatible)
	+++  CLASS FILE FORMAT VERSION: 55.0 <- n.a.
	+++  NEW INTERFACE: java.lang.constant.Constable
	+++  NEW INTERFACE: java.lang.Comparable
	+++  NEW INTERFACE: java.io.Serializable
	+++  NEW SUPERCLASS: java.lang.Enum
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory TRANSLOG
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory SEGMENTS
	+++  NEW METHOD: PUBLIC(+) java.lang.String getName()
	+++  NEW METHOD: PUBLIC(+) boolean isSupportedDataType(org.opensearch.index.remote.RemoteStoreDataEnums$DataType)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory valueOf(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory[] values()
+++  NEW ENUM: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType  (compatible)
	+++  CLASS FILE FORMAT VERSION: 55.0 <- n.a.
	+++  NEW INTERFACE: java.lang.constant.Constable
	+++  NEW INTERFACE: java.lang.Comparable
	+++  NEW INTERFACE: java.io.Serializable
	+++  NEW SUPERCLASS: java.lang.Enum
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType LOCK_FILES
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType DATA
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType METADATA
	+++  NEW METHOD: PUBLIC(+) java.lang.String getName()
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType valueOf(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStoreDataEnums$DataType[] values()
+++  NEW ENUM: PUBLIC(+) ABSTRACT(+) org.opensearch.index.remote.RemoteStorePathType  (compatible)
	+++  CLASS FILE FORMAT VERSION: [55](https://github.com/opensearch-project/OpenSearch/actions/runs/8844258476/job/24285816434?pr=13405#step:5:56).0 <- n.a.
	+++  NEW INTERFACE: java.lang.constant.Constable
	+++  NEW INTERFACE: java.lang.Comparable
	+++  NEW INTERFACE: java.io.Serializable
	+++  NEW SUPERCLASS: java.lang.Enum
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStorePathType FIXED
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStorePathType HASHED_PREFIX
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) java.lang.String NAME
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStorePathType parseString(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.common.blobstore.BlobPath path(org.opensearch.common.blobstore.BlobPath, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory, org.opensearch.index.remote.RemoteStoreDataEnums$DataType)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStorePathType valueOf(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStorePathType[] values()
***! MODIFIED CLASS: PUBLIC org.opensearch.index.store.lockmanager.RemoteStoreLockManagerFactory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) org.opensearch.index.store.lockmanager.RemoteStoreLockManager newLockManager(java.lang.String, java.lang.String, java.lang.String)
		---  REMOVED EXCEPTION: java.io.IOException
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.store.lockmanager.RemoteStoreMetadataLockManager newLockManager(org.opensearch.repositories.RepositoriesService, java.lang.String, java.lang.String, java.lang.String)
		---  REMOVED EXCEPTION: java.io.IOException
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.store.lockmanager.RemoteStoreLockManager newLockManager(java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathType)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.store.lockmanager.RemoteStoreMetadataLockManager newLockManager(org.opensearch.repositories.RepositoriesService, java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathType)
***! MODIFIED CLASS: PUBLIC FINAL org.opensearch.index.store.RemoteSegmentStoreDirectory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) void remoteDirectoryCleanup(org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory, java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) void remoteDirectoryCleanup(org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory, java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathType)
***! MODIFIED CLASS: PUBLIC org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) org.apache.lucene.store.Directory newDirectory(java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId)
		---  REMOVED EXCEPTION: java.io.IOException
	+++  NEW METHOD: PUBLIC(+) org.apache.lucene.store.Directory newDirectory(java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathType)
		+++  NEW EXCEPTION: java.io.IOException