- OpenSearch SDK for Java Developer Guide
In general, running and using an extension can be broken down into the following steps:
- Start the extension:
- Clone the OpenSearch SDK for Java repository.
- Run your own extension or the sample Hello World extension:
- Start OpenSearch:
- Use the extension:
Note: You need to first start the extension or extensions and then start OpenSearch.
This tutorial uses the sample Hello World extension included in the opensearch-sdk-java
repository.
To start the extension, you need to first clone the OpenSearch SDK for Java repository and then run the extension.
Fork the OpenSearch SDK for Java repository and clone it locally using the following command:
git clone https://github.com/<your username>/opensearch-sdk-java.git
Navigate to the directory to which you cloned the OpenSearch SDK for Java repository.
You can run the sample Hello World extension using the helloWorld
task:
./gradlew helloWorld
Bound addresses will then be logged to the terminal:
[main] INFO transportservice.TransportService - publish_address {127.0.0.1:3333}, bound_addresses {[::1]:3333}, {127.0.0.1:3333}
[main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555}
This is not required.
- Uncomment the SSL settings from resources/sample/helloworld-settings.yml:
ssl.transport.enabled: true
ssl.transport.pemcert_filepath: certs/extension-01.pem
ssl.transport.pemkey_filepath: certs/extension-01-key.pem
ssl.transport.pemtrustedcas_filepath: certs/root-ca.pem
ssl.transport.enforce_hostname_verification: false
path.home: <path/to/extension>
- Follow the instructions in CERTIFICATE_GENERATION to generate the certificates.
- Run the extension using
./gradlew run
.
Follow these steps to start OpenSearch:
Fork the OpenSearch repository and clone it locally using the following command:
git clone https://github.com/<your username>/OpenSearch.git
Extensions are experimental since OpenSearch 2.9, so you must enable them either before or when you run OpenSearch. You can enable the feature flag using one of the following options.
Add the experimental feature system property to gradle/run.gradle
:
testClusters {
runTask {
testDistribution = 'archive'
if (numZones > 1) numberOfZones = numZones
if (numNodes > 1) numberOfNodes = numNodes
systemProperty 'opensearch.experimental.feature.extensions.enabled', 'true'
}
}
Add the experimental feature flag as a command line argument:
./bin/opensearch -E opensearch.experimental.feature.extensions.enabled=true
when running from a local distribution
Enable the experimental feature flag by setting it to true
in opensearch.yml
:
cd
to your local distribution build for OpenSearch.cd
into the OpenSearchconfig
folder and openopensearch.yml
in your local editor.- Search for
opensearch.experimental.feature.extensions.enabled
, uncomment it, and set it totrue
. - Run OpenSearch using
./bin/opensearch
when running from a local distribution.
You can run OpenSearch either from a compiled binary or from source code.
If you closed OpenSearch as above, use gradle
to run it.
- Run
./gradlew run
to start OpenSearch. - Send the below sample REST API to initialize an extension using settings in hello.json.
curl -XPOST "localhost:9200/_extensions/initialize" -H "Content-Type:application/json" --data @src/main/java/org/opensearch/sdk/sample/helloworld/hello.json
You can also build OpenSearch with ./gradlew assemble
or download a released version and start it using ./bin/opensearch
.
Note: If the Security plugin is initialized in OpenSearch, use admin credentials to send extension initialization request.
In response to the REST /initialize
request, ExtensionsManager
discovers the extension listening on a predefined port and executes the TCP handshake protocol to establish a data transfer connection. Then OpenSearch sends a request to the OpenSearch SDK for Java and, upon acknowledgment, the extension responds with its name. This name is logged in the terminal where OpenSearch is running:
[2022-06-16T21:30:18,857][INFO ][o.o.t.TransportService ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}
[2022-06-16T21:30:18,978][INFO ][o.o.t.TransportService ] [runTask-0] Action: internal:transport/handshake
[2022-06-16T21:30:18,989][INFO ][o.o.t.TransportService ] [runTask-0] TransportService:sendRequest action=internal:discovery/extensions
[2022-06-16T21:30:18,989][INFO ][o.o.t.TransportService ] [runTask-0] Action: internal:discovery/extensions
[2022-06-16T21:30:19,000][INFO ][o.o.e.ExtensionsManager] [runTask-0] received PluginResponse{examplepluginname}
The OpenSearch SDK terminal also logs all requests and responses it receives from OpenSearch:
- TCP handshake request:
21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 45 53 00 00 00 31 00 00 00 00 00 00 00 01 08 08 |ES...1..........|
|00000010| 1e ab f3 00 00 00 1a 00 00 00 16 69 6e 74 65 72 |...........inter|
|00000020| 6e 61 6c 3a 74 63 70 2f 68 61 6e 64 73 68 61 6b |nal:tcp/handshak|
|00000030| 65 00 04 a3 8e b7 41 |e.....A |
+--------+-------------------------------------------------+----------------+
MESSAGE RECEIVED:E«internal:tcp/handshake£·A
- Extension name request/response:
21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 45 53 00 00 00 c6 00 00 00 00 00 00 00 03 00 08 |ES..............|
|00000010| 2d c7 23 00 00 00 21 00 00 00 1d 69 6e 74 65 72 |-.#...!....inter|
|00000020| 6e 61 6c 3a 64 69 73 63 6f 76 65 72 79 2f 65 78 |nal:discovery/ex|
|00000030| 74 65 6e 73 69 6f 6e 73 00 00 0e 6e 6f 64 65 5f |tensions...node_|
|00000040| 65 78 74 65 6e 73 69 6f 6e 16 51 53 74 39 6f 4b |extension.QSt9oK|
|00000050| 58 46 54 53 57 71 67 58 34 62 6b 56 6a 47 2d 51 |XFTSWqgX4bkVjG-Q|
|00000060| 09 31 32 37 2e 30 2e 30 2e 31 09 31 32 37 2e 30 |.127.0.0.1.127.0|
|00000070| 2e 30 2e 31 04 7f 00 00 01 09 31 32 37 2e 30 2e |.0.1......127.0.|
|00000080| 30 2e 31 00 00 11 b4 00 04 0f 63 6c 75 73 74 65 |0.1.......cluste|
|00000090| 72 5f 6d 61 6e 61 67 65 72 01 6d 00 04 64 61 74 |r_manager.m..dat|
|000000a0| 61 01 64 01 06 69 6e 67 65 73 74 01 69 00 15 72 |a.d..ingest.i..r|
|000000b0| 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 6c |emote_cluster_cl|
|000000c0| 69 65 6e 74 01 72 00 a3 8e b7 41 00 |ient.r....A. |
+--------+-------------------------------------------------+----------------+
MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1 127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A
21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B
21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] received request
21:30:18.996 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - register 2 [transport] [internal:discovery/extensions] []
21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - unregister task for id: 2
21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B
21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B
+-------------------------------------------------+
| 0 1 2 3 4 5 6 7 8 9 a b c d e f |
+--------+-------------------------------------------------+----------------+
|00000000| 45 53 00 00 00 2a 00 00 00 00 00 00 00 03 01 08 |ES...*..........|
|00000010| 2d c7 23 00 00 00 02 00 00 16 4e 61 6d 65 64 57 |-.#.......exampl|
|00000020| 72 69 74 65 61 62 6c 65 52 65 67 69 73 74 72 79 |epluginname |
+--------+-------------------------------------------------+----------------+
21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH
21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] sent response
The following request is configured to be handled by the sample HelloWorldExtension
(note that its matching uniqueId
is hello-world-java
):
curl -X GET localhost:9200/_extensions/_hello-world-java/hello
Before you write your own extension, read through the design documentation to learn about extension architecture and class hierarchy. Then follow this guide to develop your own extension. For an example, see the sample Hello World extension in the org.opensearch.sdk.sample.helloworld
package.
Refer to the following sections for information about post-development tasks.
To run an extension that uses the SDK, use ./gradlew run
on that extension.
Until we publish this repo to the Maven Central Repository, publishing to the Maven local repository is how extensions (outside of sample packages) import the artifacts:
./gradlew publishToMavenLocal
Use the following command to run tests:
./gradlew clean build integTest
For information about launching and debugging from an IDE in OpenSearch, see this document
In opensearch-sdk-java
, navigate to build/distributions
. Look for the tarball in the form opensearch-sdk-java-1.0.0-SNAPSHOT.tar
. If there is no such tarball, use the following command to create one:
./gradlew clean && ./gradlew build
Once the tarball is generated, navigate to /src/test/resources/sample
and look for extension-settings.yml
. If the file is not present, create it.
The tarball is generated in /build/distributions
. To run the artifact (the tarball), use the following command:
tar -xvf opensearch-sdk-java-1.0.0-SNAPSHOT.tar
The artifact will include extension settings for the sample Hello World extension on the class path under the path /sample/extension-settings.yml
:
extensionName: hello-world
hostAddress: 127.0.0.1
hostPort: 4532
opensearchAddress: 127.0.0.1
opensearchPort: 9200
You can optionally add routeNamePrefix:
as a value to the yml. This setting allows you to prefix all your registered NamedRoute names.
The value must be alphanumeric and can contain _
in the name.
Start the sample extension with ./bin/opensearch-sdk-java
To learn how to submit your changes, see CONTRIBUTING.