Merge remote-tracking branch 'sgup432/ehcache_disk_integ' into keysto…

…re-squashed

CHANGELOG.md

@@ -118,6 +118,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Request level coordinator slow logs ([#10650](https://github.com/opensearch-project/OpenSearch/pull/10650))
 - Add template snippets support for field and target_field in KV ingest processor ([#10040](https://github.com/opensearch-project/OpenSearch/pull/10040))
 - Allowing pipeline processors to access index mapping info by passing ingest service ref as part of the processor factory parameters ([#10307](https://github.com/opensearch-project/OpenSearch/pull/10307))
+- Add experimental SIMD implementation of B-tree to round down dates ([#11194](https://github.com/opensearch-project/OpenSearch/issues/11194))
 - Make number of segment metadata files in remote segment store configurable ([#11329](https://github.com/opensearch-project/OpenSearch/pull/11329))
 - Allow changing number of replicas of searchable snapshot index ([#11317](https://github.com/opensearch-project/OpenSearch/pull/11317))
 - Adding slf4j license header to LoggerMessageFormat.java ([#11069](https://github.com/opensearch-project/OpenSearch/pull/11069))
@@ -129,7 +130,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add match_only_text field that is optimized for storage by trading off positional queries performance ([#6836](https://github.com/opensearch-project/OpenSearch/pull/11039))
 - Add copy ingest processor ([#11870](https://github.com/opensearch-project/OpenSearch/pull/11870))
 - Introduce new feature flag "WRITEABLE_REMOTE_INDEX" to gate the writeable remote index functionality ([#11717](https://github.com/opensearch-project/OpenSearch/pull/11170))
+- [Tiered caching] Integrating ehcache as a disk cache option ([#11874](https://github.com/opensearch-project/OpenSearch/pull/11874))
 - Bump OpenTelemetry from 1.32.0 to 1.34.1 ([#11891](https://github.com/opensearch-project/OpenSearch/pull/11891))
+- Support index level allocation filtering for searchable snapshot index ([#11563](https://github.com/opensearch-project/OpenSearch/pull/11563))
+- Add `org.opensearch.rest.MethodHandlers` and `RestController#getAllHandlers` ([11876](https://github.com/opensearch-project/OpenSearch/pull/11876))
 
 ### Dependencies
 - Bumps jetty version to 9.4.52.v20230823 to fix GMS-2023-1857 ([#9822](https://github.com/opensearch-project/OpenSearch/pull/9822))
@@ -172,6 +176,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Bump `lycheeverse/lychee-action` from 1.8.0 to 1.9.1 ([#11795](https://github.com/opensearch-project/OpenSearch/pull/11795), [#11887](https://github.com/opensearch-project/OpenSearch/pull/11887))
 - Bump `Lucene` from 9.8.0 to 9.9.1 ([#11421](https://github.com/opensearch-project/OpenSearch/pull/11421))
 - Bump `com.networknt:json-schema-validator` from 1.0.86 to 1.1.0 ([#11886](https://github.com/opensearch-project/OpenSearch/pull/11886))
+- Bump `com.google.api:gax-httpjson` from 0.103.1 to 2.39.0 ([#11794](https://github.com/opensearch-project/OpenSearch/pull/11794))
+- Bump `com.google.oauth-client:google-oauth-client` from 1.34.1 to 1.35.0 ([#11960](https://github.com/opensearch-project/OpenSearch/pull/11960))
+- Bump `com.diffplug.spotless` from 6.23.2 to 6.24.0 ([#11962](https://github.com/opensearch-project/OpenSearch/pull/11962))
 
 ### Changed
 - Mute the query profile IT with concurrent execution ([#9840](https://github.com/opensearch-project/OpenSearch/pull/9840))
@@ -202,6 +209,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Use slice_size == shard_size heuristic in terms aggs for concurrent segment search and properly calculate the doc_count_error ([#11732](https://github.com/opensearch-project/OpenSearch/pull/11732))
 - Added Support for dynamically adding SearchRequestOperationsListeners with SearchRequestOperationsCompositeListenerFactory ([#11526](https://github.com/opensearch-project/OpenSearch/pull/11526))
 - Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings ([#11890](https://github.com/opensearch-project/OpenSearch/pull/11890))
+- Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase ([#11877](https://github.com/opensearch-project/OpenSearch/pull/11877))
 
 ### Deprecated
 
@@ -224,13 +232,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Fix remote shards balancer and remove unused variables ([#11167](https://github.com/opensearch-project/OpenSearch/pull/11167))
 - Fix parsing of flat object fields with dots in keys ([#11425](https://github.com/opensearch-project/OpenSearch/pull/11425))
 - Fix bug where replication lag grows post primary relocation ([#11238](https://github.com/opensearch-project/OpenSearch/pull/11238))
-- Fix noop_update_total metric in indexing stats cannot be updated by bulk API ([#11485](https://github.com/opensearch-project/OpenSearch/pull/11485))
+- Fix noop_update_total metric in indexing stats cannot be updated by bulk API ([#11485](https://github.com/opensearch-project/OpenSearch/pull/11485),[#11917](https://github.com/opensearch-project/OpenSearch/pull/11917))
 - Fix for stuck update action in a bulk with `retry_on_conflict` property ([#11152](https://github.com/opensearch-project/OpenSearch/issues/11152))
 - Fix template setting override for replication type ([#11417](https://github.com/opensearch-project/OpenSearch/pull/11417))
 - Fix Automatic addition of protocol broken in #11512 ([#11609](https://github.com/opensearch-project/OpenSearch/pull/11609))
 - Fix issue when calling Delete PIT endpoint and no PITs exist ([#11711](https://github.com/opensearch-project/OpenSearch/pull/11711))
 - Fix tracing context propagation for local transport instrumentation ([#11490](https://github.com/opensearch-project/OpenSearch/pull/11490))
 - Fix parsing of single line comments in `lang-painless` ([#11815](https://github.com/opensearch-project/OpenSearch/issues/11815))
+- Fix memory leak issue in ReorganizingLongHash ([#11953](https://github.com/opensearch-project/OpenSearch/issues/11953))
 
 ### Security
 

benchmarks/build.gradle

@@ -84,3 +84,45 @@ spotless {
     targetExclude 'src/main/generated/**/*.java'
   }
 }
+
+if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
+  // Add support for incubator modules on supported Java versions.
+  run.jvmArgs += ['--add-modules=jdk.incubator.vector']
+  run.classpath += files(jar.archiveFile)
+  run.classpath -= sourceSets.main.output
+  evaluationDependsOn(':libs:opensearch-common')
+
+  sourceSets {
+    java20 {
+      java {
+        srcDirs = ['src/main/java20']
+      }
+    }
+  }
+
+  configurations {
+    java20Implementation.extendsFrom(implementation)
+  }
+
+  dependencies {
+    java20Implementation sourceSets.main.output
+    java20Implementation project(':libs:opensearch-common').sourceSets.java20.output
+    java20AnnotationProcessor "org.openjdk.jmh:jmh-generator-annprocess:$versions.jmh"
+  }
+
+  compileJava20Java {
+    targetCompatibility = JavaVersion.VERSION_20
+    options.compilerArgs.addAll(["-processor", "org.openjdk.jmh.generators.BenchmarkProcessor"])
+  }
+
+  jar {
+    metaInf {
+      into 'versions/20'
+      from sourceSets.java20.output
+    }
+    manifest.attributes('Multi-Release': 'true')
+  }
+
+  // classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
+  disableTasks('forbiddenApisJava20')
+}

benchmarks/src/main/java/org/opensearch/common/round/RoundableBenchmark.java

@@ -21,7 +21,6 @@
 import org.openjdk.jmh.infra.Blackhole;
 
 import java.util.Random;
-import java.util.function.Supplier;
 
 @Fork(value = 3)
 @Warmup(iterations = 3, time = 1)
@@ -83,17 +82,17 @@ public static class Options {
             "256" })
         public Integer size;
 
-        @Param({ "binary", "linear" })
+        @Param({ "binary", "linear", "btree" })
         public String type;
 
         @Param({ "uniform", "skewed_edge", "skewed_center" })
         public String distribution;
 
         public long[] queries;
-        public Supplier<Roundable> supplier;
+        public RoundableSupplier supplier;
 
         @Setup
-        public void setup() {
+        public void setup() throws ClassNotFoundException {
             Random random = new Random(size);
             long[] values = new long[size];
             for (int i = 1; i < values.length; i++) {
@@ -128,16 +127,7 @@ public void setup() {
                     throw new IllegalArgumentException("invalid distribution: " + distribution);
             }
 
-            switch (type) {
-                case "binary":
-                    supplier = () -> new BinarySearcher(values, size);
-                    break;
-                case "linear":
-                    supplier = () -> new BidirectionalLinearSearcher(values, size);
-                    break;
-                default:
-                    throw new IllegalArgumentException("invalid type: " + type);
-            }
+            supplier = new RoundableSupplier(type, values, size);
         }
 
         private static long nextPositiveLong(Random random) {

benchmarks/src/main/java/org/opensearch/common/round/RoundableSupplier.java

@@ -0,0 +1,35 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.common.round;
+
+import java.util.function.Supplier;
+
+public class RoundableSupplier implements Supplier<Roundable> {
+    private final Supplier<Roundable> delegate;
+
+    RoundableSupplier(String type, long[] values, int size) throws ClassNotFoundException {
+        switch (type) {
+            case "binary":
+                delegate = () -> new BinarySearcher(values, size);
+                break;
+            case "linear":
+                delegate = () -> new BidirectionalLinearSearcher(values, size);
+                break;
+            case "btree":
+                throw new ClassNotFoundException("BtreeSearcher is not supported below JDK 20");
+            default:
+                throw new IllegalArgumentException("invalid type: " + type);
+        }
+    }
+
+    @Override
+    public Roundable get() {
+        return delegate.get();
+    }
+}

benchmarks/src/main/java20/org/opensearch/common/round/RoundableSupplier.java

@@ -0,0 +1,36 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.common.round;
+
+import java.util.function.Supplier;
+
+public class RoundableSupplier implements Supplier<Roundable> {
+    private final Supplier<Roundable> delegate;
+
+    RoundableSupplier(String type, long[] values, int size) {
+        switch (type) {
+            case "binary":
+                delegate = () -> new BinarySearcher(values, size);
+                break;
+            case "linear":
+                delegate = () -> new BidirectionalLinearSearcher(values, size);
+                break;
+            case "btree":
+                delegate = () -> new BtreeSearcher(values, size);
+                break;
+            default:
+                throw new IllegalArgumentException("invalid type: " + type);
+        }
+    }
+
+    @Override
+    public Roundable get() {
+        return delegate.get();
+    }
+}

build.gradle

@@ -54,7 +54,7 @@ plugins {
   id 'lifecycle-base'
   id 'opensearch.docker-support'
   id 'opensearch.global-build-info'
-  id "com.diffplug.spotless" version "6.23.2" apply false
+  id "com.diffplug.spotless" version "6.24.0" apply false
   id "org.gradle.test-retry" version "1.5.4" apply false
   id "test-report-aggregation"
   id 'jacoco-report-aggregation'

buildSrc/version.properties

@@ -72,3 +72,4 @@ resteasy          = 6.2.4.Final
 # opentelemetry dependencies
 opentelemetry         = 1.34.1
 opentelemetrysemconv  = 1.23.1-alpha
+ehcache = 3.10.8

client/sniffer/src/main/java/org/opensearch/client/sniff/Sniffer.java

@@ -305,6 +305,7 @@ public void shutdown() {
         }
     }
 
+    @SuppressWarnings("removal")
     static class SnifferThreadFactory implements ThreadFactory {
         private final AtomicInteger threadNumber = new AtomicInteger(1);
         private final String namePrefix;

distribution/archives/integ-test-zip/src/test/java/org/opensearch/test/rest/JsonLogsFormatAndParseIT.java

@@ -51,6 +51,7 @@ protected Matcher<String> nodeNameMatcher() {
         return is("integTest-0");
     }
 
+    @SuppressWarnings("removal")
     @Override
     protected BufferedReader openReader(Path logFile) {
         assumeFalse("Skipping test because it is being run against an external cluster.",

libs/common/build.gradle

@@ -43,3 +43,64 @@ tasks.named('forbiddenApisMain').configure {
   // TODO: Need to decide how we want to handle for forbidden signatures with the changes to server
   replaceSignatureFiles 'jdk-signatures'
 }
+
+// Add support for incubator modules on supported Java versions.
+if (BuildParams.runtimeJavaVersion >= JavaVersion.VERSION_20) {
+  sourceSets {
+    java20 {
+      java {
+        srcDirs = ['src/main/java20']
+      }
+    }
+  }
+
+  configurations {
+    java20Implementation.extendsFrom(implementation)
+  }
+
+  dependencies {
+    java20Implementation sourceSets.main.output
+  }
+
+  compileJava20Java {
+    targetCompatibility = JavaVersion.VERSION_20
+    options.compilerArgs += ['--add-modules', 'jdk.incubator.vector']
+    options.compilerArgs -= '-Werror' // use of incubator modules is reported as a warning
+  }
+
+  jar {
+    metaInf {
+      into 'versions/20'
+      from sourceSets.java20.output
+    }
+    manifest.attributes('Multi-Release': 'true')
+  }
+
+  tasks.withType(Test).configureEach {
+    // Relying on the convention for Test.classpath in custom Test tasks has been deprecated
+    // and scheduled to be removed in Gradle 9.0. Below lines are added from the migration guide:
+    // https://docs.gradle.org/8.5/userguide/upgrading_version_8.html#test_task_default_classpath
+    testClassesDirs = testing.suites.test.sources.output.classesDirs
+    classpath = testing.suites.test.sources.runtimeClasspath
+
+    // Adds the multi-release JAR to the classpath when executing tests.
+    // This allows newer sources to be picked up at test runtime (if supported).
+    classpath += files(jar.archiveFile)
+    // Removes the "main" sources from the classpath to avoid JarHell problems as
+    // the multi-release JAR already contains those classes.
+    classpath -= sourceSets.main.output
+  }
+
+  tasks.register('roundableSimdTest', Test) {
+    group 'verification'
+    include '**/RoundableTests.class'
+    systemProperty 'opensearch.experimental.feature.simd.rounding.enabled', 'forced'
+  }
+
+  check.dependsOn(roundableSimdTest)
+
+  forbiddenApisJava20 {
+    failOnMissingClasses = false
+    ignoreSignaturesOfMissingClasses = true
+  }
+}