Alerts not working after upgrade to OS 2.15.0

[gregecslo]

Hi.

After upgrade from Elasticsearch to Opensearch, my alerts stopped working. Tried also with Opensearch 2.16.0 same thing (also 2.16.0 has issues so I reverted to 2.15.0).

I found out what went wrong, after I added field in condition then it works on both opensearch versions (2.15 and 2.16), see below image.

I believe this is a bug, field could also be blank, and it worked just fine on Elasticsearch 7.10.2 ...

• Graylog Version: 6.0.5
• OpenSearch Version: 2.15.0
• MongoDB Version: 7.X
• Operating System: Ubuntu 22.04

[drewmiranda-gl]

There is an issue with OpenSearch 2.16, see #20119 . Though there is a work around: opensearch-project/OpenSearch#15169 (comment)

We've not received reports of issues with 2.15 though.

Do you have any applicable messages in Graylog's server.log? If you manually execute the search query configured for the event for the specific time range that you expected this to trigger, do you get results as expected?

Also to clarify, are you saying the bug is that an event will fail if an aggregation is used but the field is left empty?

I can confirm this works as I have several alerts that are configured this way (Graylog 6.0.5, OpenSearch 2.15)

The only other outstanding issue I'm aware of with alerting is https://github.com/Graylog2/graylog-plugin-enterprise/issues/7588 though not sure that is applicable here. Curious if there is anything interesting in your logs.

[gregecslo]

Yes all works ok if field is not empty. Replay search displays results just fine.

…

________________________________ From: Drew Miranda ***@***.***> Sent: Thursday, August 29, 2024 20:19 To: Graylog2/graylog2-server Cc: Grega; Author Subject: Re: [Graylog2/graylog2-server] Alerts not working after upgrade to OS 2.15.0 (Issue #20294) There is an issue with OpenSearch 2.16, see #20119<#20119> . Though there is a work around: opensearch-project/OpenSearch#15169 (comment)<opensearch-project/OpenSearch#15169 (comment)> We've not received reports of issues with 2.15 though. Do you have any applicable messages in Graylog's server.log? If you manually execute the search query configured for the event for the specific time range that you expected this to trigger, do you get results as expected? Also to clarify, are you saying the bug is that an event will fail if an aggregation is used but the field is left empty? I can confirm this works as I have several alerts that are configured this way (Graylog 6.0.5, OpenSearch 2.15) image.png (view on web)<https://github.com/user-attachments/assets/7789d9b9-9456-4fe6-a6bb-39c4046f4394> image.png (view on web)<https://github.com/user-attachments/assets/445e96f8-f95d-447b-bb5f-ffaff4dd86a5> The only other outstanding issue I'm aware of with alerting is Graylog2/graylog-plugin-enterprise#7588<https://github.com/Graylog2/graylog-plugin-enterprise/issues/7588> though not sure that is applicable here. Curious if there is anything interesting in your logs. — Reply to this email directly, view it on GitHub<#20294 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BDPHP3GJYOOSFETE7MFIHS3ZT5QZXAVCNFSM6AAAAABNJUWTNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJYGU2TOMJQGU>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[gregecslo]

Checked logs as well.
Nothing there.

Basically this is working:

Create Events if
count(@timestamp) > 0

And this is NOT working

Create Events if
count() > 0

When I used Elasticsearch everything was just fine. When I migrated to OS 2.15.0 this started to happen and it`s definitly a bug, because it happens on 2 instances of Graylog which are totally unrelated and on 2 different locations.
One is using supported 2.15.0 and other 2.16.0. Both same Graylog 6.0.5 versions and both with SAME problems AFTER I migrated to OpenSearch.

[dennisoelkers]

Hey @gregecslo,

I just tried to reproduce the issue with OS 2.16.0 and a count() > 0 event definition and it works fine for me. Is everything okay with the streams you are ingesting into? There must be something else related causing this.

[github-actions]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.