[BUG] 503 Error encountered during the upgrade of Logstash from version 8.10.4 to 8.12.1

[RamyaSaba]

Hi,

I recently performed the first upgrade of Logstash from version 8.10.4 to 8.12.1 and encountered an error during the process. It's important to note that this error only occurs during the initial upgrade and not during the installation process.

"log":{"message":"Failed to install template {:message=>"Got response code '503' contacting OpenSearch at URL 'https://opensearch:9200/_template/logstash'", :exception=>LogStash::Outputs::OpenSearch::HttpClient::Pool::BadResponseCodeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/manticore_adapter.rb:181:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:272:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:259:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:348:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:258:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client/pool.rb:266:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:393:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:398:in `template_exists?'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/http_client.rb:78:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/template_manager.rb:37:in `install'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch/template_manager.rb:25:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:419:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:254:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/outputs/opensearch.rb:231:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-opensearch-2.0.2-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}"}}

Versions:

Logstash: 8.12.1
OpenSearch: 2.11.0
This error doesn't come up on installation or further upgrades.

{"cluster_name":"test","status":"yellow","timed_out":false,"number_of_nodes":3,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":3,"active_shards":3,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":75.0}

curl -k -u<user>:<password> "http://localhost:9200/_cluster/health"
{
  "name" : "test",
  "cluster_name" : "test",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_snapshot" : false,
    "lucene_version" : "9.7.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Query:

Are there any specific steps or configurations recommended for handling the upgrade process?
Could there be a compatibility issue between Logstash 8.12.1 and OpenSearch 2.11.0 that manifests uniquely during the initial upgrade?
Any guidance or insights on resolving this issue would be greatly appreciated.

Thank you!

[dblock]

Looks similar to #145 and opensearch-project/opensearch-devops#85 which seems to have a solution, see if that helps?

[RamyaSaba]

The above solutions are similar but doesn't help me much with the error I am encountering.
Could you please provide insights into why the "Failed to install template" error occurs during the upgrade of Logstash version? Specifically, what conditions or scenarios might trigger this error?

What steps can be taken to troubleshoot and resolve this issue?
Are there any configurations or settings within Logstash or OpenSearch that should be checked or adjusted to mitigate this error?

[dblock]

@RamyaSaba I really don't know that much about logstash, but please do post here if you find answers to any of these.