Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Backport 2.x] Introduce remote store hash algorithm in customData in IndexMetadata #13407

Merged
merged 1 commit into from
Apr 26, 2024
Merged

[Backport 2.x] Introduce remote store hash algorithm in customData in IndexMetadata #13407

merged 1 commit into from
Apr 26, 2024

Conversation

opensearch-trigger-bot[bot]
Copy link
Contributor

Backport 170ea7a from #12986.

@github-actions
Introduce remote store hash algo in customData in IndexMetadata (#12986)
02a62b3 
Signed-off-by: Ashish Singh <[email protected]>
(cherry picked from commit 170ea7a)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
@github-actions GitHub Actions
Copy link
Contributor

✅ Gradle check result for 02a62b3: SUCCESS

@ashking94
Copy link
Member

The breaking change failure is for unreleased code (this code will be released for the first time in 2.14) -

***! MODIFIED CLASS: PUBLIC FINAL org.opensearch.index.IndexSettings  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy getRemoteStorePathStrategy()
	---! REMOVED METHOD: PUBLIC(-) org.opensearch.index.remote.RemoteStorePathType getRemoteStorePathType()
---! REMOVED ENUM: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory  (class removed)
	---  CLASS FILE FORMAT VERSION: n.a. <- 55.0
	---! REMOVED INTERFACE: java.lang.constant.Constable
	---! REMOVED INTERFACE: java.lang.Comparable
	---! REMOVED INTERFACE: java.io.Serializable
	---! REMOVED SUPERCLASS: java.lang.Enum
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory TRANSLOG
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory SEGMENTS
	---! REMOVED METHOD: PUBLIC(-) java.lang.String getName()
	---! REMOVED METHOD: PUBLIC(-) boolean isSupportedDataType(org.opensearch.index.remote.RemoteStoreDataEnums$DataType)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory valueOf(java.lang.String)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory[] values()
---! REMOVED ENUM: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType  (class removed)
	---  CLASS FILE FORMAT VERSION: n.a. <- 55.0
	---! REMOVED INTERFACE: java.lang.constant.Constable
	---! REMOVED INTERFACE: java.lang.Comparable
	---! REMOVED INTERFACE: java.io.Serializable
	---! REMOVED SUPERCLASS: java.lang.Enum
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType LOCK_FILES
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType DATA
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType METADATA
	---! REMOVED METHOD: PUBLIC(-) java.lang.String getName()
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType valueOf(java.lang.String)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStoreDataEnums$DataType[] values()
+++  NEW ENUM: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreEnums$DataCategory  (compatible)
	+++  CLASS FILE FORMAT VERSION: 55.0 <- n.a.
	+++  NEW INTERFACE: java.lang.constant.Constable
	+++  NEW INTERFACE: java.lang.Comparable
	+++  NEW INTERFACE: java.io.Serializable
	+++  NEW SUPERCLASS: java.lang.Enum
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreEnums$DataCategory TRANSLOG
	+++  NEW FIELD: PUBLIC(+) STATIC(+) FINAL(+) org.opensearch.index.remote.RemoteStoreEnums$DataCategory SEGMENTS
	+++  NEW METHOD: PUBLIC(+) java.lang.String getName()
	+++  NEW SUPERCLASS: java.lang.Object
	+++  NEW CONSTRUCTOR: PUBLIC(+) RemoteStorePathStrategy$PathInput(org.opensearch.common.blobstore.BlobPath, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStoreEnums$DataCategory, org.opensearch.index.remote.RemoteStoreEnums$DataType)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder builder()
+++  NEW CLASS: PUBLIC(+) STATIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder  (not serializable)
	+++  CLASS FILE FORMAT VERSION: 55.0 <- n.a.
	+++  NEW SUPERCLASS: java.lang.Object
	+++  NEW CONSTRUCTOR: PUBLIC(+) RemoteStorePathStrategy$PathInput$Builder()
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder basePath(org.opensearch.common.blobstore.BlobPath)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput build()
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder dataCategory(org.opensearch.index.remote.RemoteStoreEnums$DataCategory)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder dataType(org.opensearch.index.remote.RemoteStoreEnums$DataType)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder indexUUID(java.lang.String)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.remote.RemoteStorePathStrategy$PathInput$Builder shardId(java.lang.String)
---! REMOVED ENUM: PUBLIC(-) ABSTRACT(-) org.opensearch.index.remote.RemoteStorePathType  (class removed)
	---  CLASS FILE FORMAT VERSION: n.a. <- 55.0
	---! REMOVED INTERFACE: java.lang.constant.Constable
	---! REMOVED INTERFACE: java.lang.Comparable
	---! REMOVED INTERFACE: java.io.Serializable
	---! REMOVED SUPERCLASS: java.lang.Enum
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStorePathType FIXED
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) org.opensearch.index.remote.RemoteStorePathType HASHED_PREFIX
	---! REMOVED FIELD: PUBLIC(-) STATIC(-) FINAL(-) java.lang.String NAME
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStorePathType parseString(java.lang.String)
	---! REMOVED METHOD: PUBLIC(-) org.opensearch.common.blobstore.BlobPath path(org.opensearch.common.blobstore.BlobPath, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStoreDataEnums$DataCategory, org.opensearch.index.remote.RemoteStoreDataEnums$DataType)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStorePathType valueOf(java.lang.String)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.remote.RemoteStorePathType[] values()
***! MODIFIED CLASS: PUBLIC org.opensearch.index.store.lockmanager.RemoteStoreLockManagerFactory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) org.opensearch.index.store.lockmanager.RemoteStoreLockManager newLockManager(java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathType)
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) org.opensearch.index.store.lockmanager.RemoteStoreMetadataLockManager newLockManager(org.opensearch.repositories.RepositoriesService, java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathType)
	+++  NEW METHOD: PUBLIC(+) org.opensearch.index.store.lockmanager.RemoteStoreLockManager newLockManager(java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathStrategy)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) org.opensearch.index.store.lockmanager.RemoteStoreMetadataLockManager newLockManager(org.opensearch.repositories.RepositoriesService, java.lang.String, java.lang.String, java.lang.String, org.opensearch.index.remote.RemoteStorePathStrategy)
***! MODIFIED CLASS: PUBLIC FINAL org.opensearch.index.store.RemoteSegmentStoreDirectory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) STATIC(-) void remoteDirectoryCleanup(org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory, java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathType)
	+++  NEW METHOD: PUBLIC(+) STATIC(+) void remoteDirectoryCleanup(org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory, java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathStrategy)
***! MODIFIED CLASS: PUBLIC org.opensearch.index.store.RemoteSegmentStoreDirectoryFactory  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	---! REMOVED METHOD: PUBLIC(-) org.apache.lucene.store.Directory newDirectory(java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathType)
		---  REMOVED EXCEPTION: java.io.IOException
	+++  NEW METHOD: PUBLIC(+) org.apache.lucene.store.Directory newDirectory(java.lang.String, java.lang.String, org.opensearch.core.index.shard.ShardId, org.opensearch.index.remote.RemoteStorePathStrategy)
		+++  NEW EXCEPTION: java.io.IOException
===* UNCHANGED CLASS: PUBLIC org.opensearch.indices.IndicesService  (not serializable)
	===  CLASS FILE FORMAT VERSION: 55.0 <- 55.0
	===* UNCHANGED FIELD: PUBLIC STATIC FINAL org.opensearch.common.settings.Setting<org.opensearch.index.remote.RemoteStoreEnums$PathType>(<- <org.opensearch.index.remote.RemoteStorePathType>) CLUSTER_REMOTE_STORE_PATH_PREFIX_TYPE_SETTING

@ashking94
Copy link
Member 

./gradlew assemble                                                                                                                                                           ✔ 
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.7
  OS Info               : Mac OS X 14.3.1 (aarch64)
  JDK Version           : 17 (Amazon Corretto JDK)
  JAVA_HOME             : /Users/ssashish/Library/Java/JavaVirtualMachines/corretto-17.0.7/Contents/Home
  Random Testing Seed   : 78B7407A860D6F2A
  In FIPS 140 mode      : false
=======================================

> Task :server:compileJava
Note: Processing Log4j annotations
Note: Annotations processed
Note: Processing OpenSearch Api annotations
Note: Processing Log4j annotations
Note: No elements to process
Note: Processing OpenSearch Api annotations
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :modules:aggs-matrix-stats:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/modules/aggs-matrix-stats/src/main/java/org/opensearch/search/aggregations/matrix/MatrixAggregationPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:parent-join:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:geo:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/modules/geo/src/main/java/org/opensearch/geo/GeoModulePlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :modules:reindex:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:lang-expression:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:systemd:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/modules/systemd/src/main/java/org/opensearch/systemd/SystemdPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:percolator:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:transport-netty4:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :distribution:tools:plugin-cli:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/distribution/tools/plugin-cli/src/main/java/org/opensearch/plugins/InstallPluginCommand.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :example-plugins:custom-suggester:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/plugins/examples/custom-suggester/src/main/java/org/opensearch/example/customsuggester/CustomSuggestion.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:analysis-icu:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:discovery-ec2:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/plugins/discovery-ec2/src/main/java/org/opensearch/discovery/ec2/Ec2DiscoveryPlugin.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :modules:analysis-common:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:query-insights:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/plugins/query-insights/src/main/java/org/opensearch/plugin/insights/rules/transport/top_queries/TransportTopQueriesAction.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :benchmarks:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/benchmarks/src/main/java/org/opensearch/benchmark/time/DateFormatterBenchmark.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :plugins:repository-azure:compileJava
Note: /Users/ssashish/dev/github/OpenSearch/plugins/repository-azure/src/main/java/org/opensearch/repositories/azure/AzureBlobStore.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :test:framework:compileJava
Note: Processing Log4j annotations
Note: Annotations processed
Note: Processing Log4j annotations
Note: No elements to process
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

> Task :qa:full-cluster-restart:compileTestJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :client:rest-high-level:compileJava
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

> Task :qa:repository-multi-version:compileTestJava
Note: /Users/ssashish/dev/github/OpenSearch/qa/repository-multi-version/src/test/java/org/opensearch/upgrades/MultiVersionRepositoryAccessIT.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

BUILD SUCCESSFUL in 4m 23s
945 actionable tasks: 353 executed, 592 up-to-date
WARNING: The following functionality has been deprecated and will be removed in the next major release of the Develocity Gradle plugin. For assistance with migration, see https://gradle.com/help/gradle-plugin-develocity-migration.
- The deprecated "gradle.enterprise.testretry.enabled" system property has been replaced by "develocity.testretry.enabled"
- The "com.gradle.enterprise" plugin has been replaced by "com.gradle.develocity"

@sachinpkale
Copy link
Member

Gradle assemble failure seems to be GitHub flow related as it is working fine on local.

@sachinpkale sachinpkale merged commit 4bea236 into 2.x Apr 26, 2024
39 of 50 checks passed
@github-actions github-actions bot deleted the backport/backport-12986-to-2.x branch April 26, 2024 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sachinpkale sachinpkale sachinpkale approved these changes

@anasalkouz anasalkouz Awaiting requested review from anasalkouz anasalkouz is a code owner

@andrross andrross Awaiting requested review from andrross andrross is a code owner

@Bukhtawar Bukhtawar Awaiting requested review from Bukhtawar Bukhtawar is a code owner

@CEHENKLE CEHENKLE Awaiting requested review from CEHENKLE CEHENKLE is a code owner

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@dbwiddis dbwiddis Awaiting requested review from dbwiddis dbwiddis is a code owner

@dreamer-89 dreamer-89 Awaiting requested review from dreamer-89

@gbbafna gbbafna Awaiting requested review from gbbafna gbbafna is a code owner

@kotwanikunal kotwanikunal Awaiting requested review from kotwanikunal kotwanikunal is a code owner

@mch2 mch2 Awaiting requested review from mch2 mch2 is a code owner

@msfroh msfroh Awaiting requested review from msfroh msfroh is a code owner

@nknize nknize Awaiting requested review from nknize nknize is a code owner

@owaiskazi19 owaiskazi19 Awaiting requested review from owaiskazi19 owaiskazi19 is a code owner

@reta reta Awaiting requested review from reta reta is a code owner

@Rishikesh1159 Rishikesh1159 Awaiting requested review from Rishikesh1159 Rishikesh1159 is a code owner

@saratvemulapalli saratvemulapalli Awaiting requested review from saratvemulapalli saratvemulapalli is a code owner

@shwetathareja shwetathareja Awaiting requested review from shwetathareja shwetathareja is a code owner

@sohami sohami Awaiting requested review from sohami sohami is a code owner

@tlfeng tlfeng Awaiting requested review from tlfeng

@VachaShah VachaShah Awaiting requested review from VachaShah VachaShah is a code owner

Assignees
No one assigned
Labels
skip-changelog
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@ashking94 @sachinpkale