opensearch output plugin compatible with logstash

[SarangBS]

I have deployed logstash and opensearch in ec2 instance

i have used bitnami helm chart for logstash

i am trying to send output from logstash to opensearch but i am unable to find a suitable plugin to do so

i need to send output to opensearch

currently the configuration of logstash is as follows

logstash:
  input: |-
    kafka
    {
      bootstrap_servers => "kafka-kafka-qa-global-kafka-brokers.unifyapps-infra.svc.cluster.local:9092"
      topics => "qa-eks-logs"
    }
  
  filter: |-
    mutate {
      rename => { "message" => "original_message" }
    }
    json {
      source => "original_message"
    }
    ruby {
      code => '
        def flatten_hash(hash, prefix="")
          flattened = {}
          hash.each do |key, value|
            if value.is_a?(Hash)
              flattened.merge!(flatten_hash(value, prefix + key + "_"))
            else
              flattened[prefix + key] = value
            end
          end
          flattened
        end

        # Flatten the parsed fields recursively
        event.to_hash.each do |key, value|
          if value.is_a?(Hash)
            flattened_fields = flatten_hash(value, key + "_")
            flattened_fields.each { |flattened_key, flattened_value|
              event.set(flattened_key, flattened_value)
            }
            # Remove the original nested field after flattening
            event.remove(key)
          end
        end
        # Convert specified fields to long type if they exist
        ["processing_ns", "status", "total_ns"].each do |field|
          if event.include?(field)
            event.set(field, event.get(field).to_i)  # Convert to long (integer)
          end
        end
      '
    }
    mutate {
      rename => { "original_message" => "message" }
    }

  output: |-
    stdout {
      codec => rubydebug   # Output for testing and debugging
    }
    elasticsearch {
      hosts => ["http://opensearch-logstash-svc.unifyapps-infra.svc.cluster.local:9200"]
      index => "logstash-logs-%{+YYYY.MM.dd}"
      ssl   => false
    }

the elasticsearch plugin is not working
the opensearch version currently in use is 2.4.0
is there any way

or is there any alternative available

i dont want to use elasticsearch since we are using opensearch for graylog

[jsvd]

Hi @SarangBS there's an Opensearch output plugin maintained by Opensearch here https://github.com/opensearch-project/logstash-output-opensearch

[SarangBS]

hey @jsvd thanks for the reply

do u have any idea how i can use it
should i have to create a helm chart out of it or install it through terminal
i am kinda new to this so i dont have much idea
the thing is i am using the bitnami helm chart for logstash link_for_helm_chart
i need to deploy it in an ec2 instance

[jsvd]

installing the plugin can be done using the Logstash's plugin manager: bin/logstash-plugin install logstash-output-opensearch.

You can either tell your provision tool to run this command, or create a custom docker image with this extra layer, or prepare a logstash tarball with the plugin already installed using https://www.elastic.co/guide/en/logstash/current/offline-plugins.html

[SarangBS]

hey @jsvd i am trying to do it through init container

this is my configuration

initContainers:

• name: install-opensearch-plugin
  image: docker.io/bitnami/logstash:latest
  command: ['sh', '-c', 'bin/logstash-plugin install logstash-output-opensearch']
  volumeMounts:
  • name: logstash-plugins
    mountPath: /bitnami/logstash/plugins

extraVolumes:

• name: logstash-plugins
  emptyDir: {} # Plugins directory

i will be adding this in the values.yaml file
will it work

[jsvd]

those are more questions about using k8s. initcontainers are containers that run before the application does, so the plugin installation will be in a separate containers from the actual application.

I apologize for the inconvenience, but this is a usage question, and should be asked at https://discuss.elastic.co.

GitHub is for coding issues and error reporting.