Help needed with filter request and debug log level for elastalert2 installed via helm chart

[Swarvenstein]

Hello.

I'm using elastalert2 in k9s and installing it via helm chart (version 2.17.0).
I have a rule in values.yaml:

rules:
  test_kafka_slack: |-
    ---
    name: test_kafka_slack
    type: frequency
    index: filebeat-*
    num_events: 1
    use_count_query: true
    realert:
      minutes: 5
    timeframe:
      hours: 24
    filter:
    - query:
        query_string:
          query: "(kubernetes.namespace:kafka) and (WARN or ERROR) and (not INFO)"
    alert:
    - "slack"
    slack_text_string: "Elasticalert2 kafka alert"
    slack_webhook_url: <url>

The main idea is to get count of all warning and error messages from k9s namespace kafka and send alert to slack if they occures. The result is very strange - I have alerts in slack, but they are not correct. I have them even if there is no such log messages in elastic (checking by this request via kibana discover) and wrong num_hits and num_matches. What am I doing wrong?

Btw, is there any way to set debug log level for elastalert2, installed via helm chart to see exact requests from elastalert2 to elasticsearch?

Thanks for help!

[jertel]

Yes, it is possible to enable debug logging using extraConfigOptions.

Your query is using lowercase and and or. These need to be UPPER case for Lucene query syntax.

Ex:

query: "(kubernetes.namespace: kafka) AND (WARN OR ERROR) AND (NOT INFO)"

[Swarvenstein]

Thanks for advice with uppercase AND and OR in query, but unfortunately it did not solve the issue with multiple false positive results.
And I wasn't able to turn verbose or debug level logging via extraConfigOptions and did not find any examples or advices how to do it, may I ask to provide me some help with this too? I'll be very appreciate.
Thanks a lot.

[jertel]

You need to provide config snippets, error messages, log files, etc so that we can help you.

[Swarvenstein]

I'm installing elastalert2 via helm chart: helm install elastalert2 -f values.yaml elastalert2/elastalert2 --version=2.17.0 -n logging --atomic --debug values.yaml have only resources definition added and only one rule, as I've placed above.
Connection to elasticsearch is defined as follow:

elasticsearch:
  # elasticsearch endpoint e.g. (svc.namespace||svc)
  host: elasticsearch-master
  # elasticsearch port
  port: 9200
  # whether or not to connect to es_host using TLS
  useSsl: "True"
  # Username if authenticating to ES with basic auth
  username: ""
  # Password if authenticating to ES with basic auth
  password: ""
  # Specifies an existing secret to be used for the ES username/password
  credentialsSecret: "elasticsearch-master-credentials"
  # The key in elasticsearch.credentialsSecret that stores the ES password
  credentialsSecretUsernameKey: "username"
  # The key in elasticsearch.credentialsSecret that stores the ES username
  credentialsSecretPasswordKey: "password"
  # whether or not to verify TLS certificates
  verifyCerts: "False"

Elastalert2 can connect to elasticsearch, creating index, making query, no errors or warnings except warning about verify certificates.
But the problem is that in kibana I'm making same query as in the configured rule and have no warning or error messages for kafka namespace, in the same time elastalert2 sending to slack various messages with different count of error messages.
I've tried to switch from use_count_query: true to full query and got some logs to slack, that are not even near the result of this query - they were from services outside of k8s cluster.
So I can't provide any errors, and can't figure out how to set up more verbose logging via helm-chart values.yaml to check anything else.

[jertel]

So I can't provide any errors, and can't figure out how to set up more verbose logging via helm-chart values.yaml to check anything else.

There must be pod logs available for ElastAlert 2. Unless you have explicitly disabled all log output (not the default behavior), that ElastAlert 2 logging output is being captured, and you need to determine where/how to retrieve it from the pod. That's step. Get the logs, find out what is in there and go from there.

[Swarvenstein]

Yes, there are pod logs of course, there are no errors there. How can I change logging level to debug from the helm chart?