Could not connect to a compatible version of Elasticsearch - Opensearch 2.9.0-1 - Logstash OutputPlugin 8.9.0

[espala]

Hello,

I set up Opensearch cluster. I use an SSL certificate in this cluster. I use 1 server both in the opensearch coordinating role and as opensearch dashboard.

I also plan to install logstash on this server and receive logs.

I downloaded all packages from the opensearch official page. My Opensearch cluster and dashboard are working healthy. But "Logstash OSS with OpenSearch Output Plugin" does not work.

I downloaded the style file below. I opened this file and put its contents into a directory named "/etc/opensearch-logstash". Then I edited my configuration files. I am trying to start the service manually.
logstash-oss-with-opensearch-output-plugin-8.9.0-linux-x64.tar.gz

But I keep getting the "Could not connect to a compatible version of Elasticsearch" error. The download page says it is compatible with "7.10.2 or lower".

I don't know what to do, I've searched all the pages I can on the internet, but I can't solve this problem, what can you suggest me?

[root@opsserver6 ]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8

[root@opsserver6 ]# yum list installed |grep opensearch
opensearch.x86_64                             2.9.0-1                                   @@System
opensearch-dashboards.x86_64                  2.9.0-1                                   @System

[root@opsserver6 ]# curl --insecure -XGET -u admin:admin "https://10.19.23.46:9200"
{
  "name" : "opsserver6",
  "cluster_name" : "central-log-cluster",
  "cluster_uuid" : "GYsnbxZdQte2Ycil7xd9Gg",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.9.0",
    "build_type" : "rpm",
    "build_hash" : "11642123kjsad12560f0ff12312312beea28433",
    "build_date" : "2023-07-18T21:22:28.183446221Z",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[root@opsserver6 opensearch-logstash]# pwd
/etc/opensearch-logstash

[root@opsserver6 opensearch-logstash]# ls
bckp  config        data               Gemfile       jdk          lib          logs           logstash-core-plugin-api  NOTICE.TXT  tools  vendor
bin   CONTRIBUTORS  dead_letter_queue  Gemfile.lock  JDK_VERSION  LICENSE.txt  logstash-core  modules                   queue       uuid

[root@opsserver6 opensearch-logstash]# cd config/

[root@opsserver6 config]# ls
opsserver6-key.pem  opsserver6.pem  jvm.options  log4j2.properties  logstash.conf  logstash.yml  old  pipelines.yml

[root@opsserver6 config]# cat logstash.yml
node.name: opsserver6
path.data: /etc/opensearch-logstash/

[root@opsserver6 config]# cat logstash.conf
input {
  beats {
    port => "5044"
  }
}

output {
  elasticsearch {
    hosts => ["https://admin:[email protected]:9200"]
    index => "filebeat-%{+YYYY.MM.dd}"
    ssl_enabled => "true"
    ssl_verification_mode => "none"
  }
}

[root@opsserver6 config]# cat pipelines.yml
- pipeline.id: main
  path.config: "/etc/opensearch-logstash/config/logstash.conf"

[root@opsserver6 config]# ../bin/logstash
Using bundled JDK: /etc/opensearch-logstash/jdk
Sending Logstash logs to /etc/opensearch-logstash/logs which is now configured via log4j2.properties
[2023-10-10T16:18:58,543][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/opensearch-logstash/config/log4j2.properties
[2023-10-10T16:18:58,548][WARN ][logstash.runner          ] The use of JAVA_HOME has been deprecated. Logstash 8.0 and later ignores JAVA_HOME and uses the bundled JDK. Running Logstash with the bundled JDK is recommended. The bundled JDK has been verified to work with each specific version of Logstash, and generally provides best performance and reliability. If you have compelling reasons for using your own JDK (organizational-specific compliance requirements, for example), you can configure LS_JAVA_HOME to use that version instead.
[2023-10-10T16:18:58,548][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.9.0", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}
[2023-10-10T16:18:58,550][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms2g, -Xmx2g, -Djava.io.tmpdir=/usr/share/logstash, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/sinatra-2.2.4/lib/sinatra/base.rb:938: warning: constant Tilt::Cache is deprecated
[2023-10-10T16:18:59,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-10-10T16:18:59,383][INFO ][org.reflections.Reflections] Reflections took 114 ms to scan 1 urls, producing 132 keys and 464 values
[2023-10-10T16:18:59,873][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-10-10T16:18:59,904][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://admin:[email protected]:9200/"]}
[2023-10-10T16:18:59,910][WARN ][logstash.outputs.elasticsearch][main] You have enabled encryption but DISABLED certificate verification, to make sure your data is secure set `ssl_verification_mode => full`
[2023-10-10T16:19:00,028][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:[email protected]:9200/]}}
[2023-10-10T16:19:00,261][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/RubyHash.java:1519:in `each'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:238:in `healthcheck!'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:376:in `update_urls'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:87:in `update_initial_urls'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:81:in `start'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client.rb:362:in `build_pool'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "org/jruby/RubyClass.java:890:in `new'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:42:in `build_client'", "/etc/opensearch-logstash/vendor/bundle/jruby/2.6.0/gems/logstash-output-elasticsearch-11.15.9-java/lib/logstash/outputs/elasticsearch.rb:300:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:69:in `register'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:237:in `block in register_plugins'", "org/jruby/RubyArray.java:1865:in `each'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:236:in `register_plugins'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:610:in `maybe_setup_out_plugins'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:249:in `start_workers'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:194:in `run'", "/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `block in start'"], "pipeline.sources"=>["/etc/opensearch-logstash/config/logstash.conf"], :thread=>"#<Thread:0x7d6e12ed@/etc/opensearch-logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-10-10T16:19:00,262][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-10-10T16:19:00,276][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2023-10-10T16:19:00,292][INFO ][logstash.runner          ] Logstash shut down.
[2023-10-10T16:19:00,297][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:790) ~[jruby.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:753) ~[jruby.jar:?]
        at etc.opensearch_minus_logstash.lib.bootstrap.environment.<main>(/etc/opensearch-logstash/lib/bootstrap/environment.rb:91) ~[?:?]

[dblock]

The error says you're using logstash-output-elasticsearch-11.15.9-java which is not logstash-output-opensearch. Your filebeat output above should be opensearch, not elasticsearch I think. There's more detailed setup in https://repost.aws/knowledge-center/opensearch-connect-filebeat-logstash. Does this help?

[espala]

I solved my problem. I realized I missed a very small detail. I was typing "elasticsearch" in the Output section. Actually, I should have corrected that section to "opensearch".

output {
  opensearch {
#  elasticsearch { # old
    hosts => ["https://admin:[email protected]:9200"]
    index => "filebeat-%{+YYYY.MM.dd}"
    ssl_enabled => "true"
    ssl_verification_mode => "none"
  }
}