"Wrong" use case for ElastAlert - alerting on historic data

[litsegaard]

Hi!

I have an ElastAlert use case which I'm struggling to solve. To give you an example of what I try to accomplish:

I have an index with, say, packetbeat data and the data in this index was indexed more than 70 days ago. Now I want to use an ElastAlert blacklist rule on that data but (of course?) EA is unable to find and trigger. I think (wrongly?) that the default time range applied by EA excludes the events of interest. If I add a timerange: days: 70 I don't get any hits - probably due to the fact that EA resorts to scrolling all records from ES in order to perform the match, and ES times out/closes the "cursor" - just my own thoughts on this. What, somewhat, confirms my reasoning is that it works if I press the "Test" button and specifies 70 days as EA then seems to make repeated scroll calls in order to return the result set.

If I perform a re-index operation, taking the packetbeat index and re-index (copy) that data to a new index and resetting the timestamp to today's date using an indexing pipeline and then perform the blacklist rule on the new index, ElastAlert works perfectly fine.

I fully appreciate the fact that the purpose of EA is to perform its duties on streaming data (mostly in real time) with a moving time window and that my use case could be a little awkward for EA to handle, but I'm a little puzzled as to why it handles the data in the new index so well? I get a feeling that EA adds a "today range filter" by default which could be the answer. Is it so?

If it is possible to tweak this in any way to accommodate my first use case I'd be extremely happy. Using my second approach might be a workaround initially but...

I hope my ramblings make sense and many thanks in advance if you've read this far!

[jertel]

Hello!

It makes sense what you're asking.

You can use the standalone scenario, specifying the --start and --end args to tell it the timeframe to search. Read more about command line parameters here: https://elastalert2.readthedocs.io/en/latest/flags.html

[litsegaard]

Thank you so much Jertel - for a while I really thought I'd gotten my wires crossed :) If I understand you correctly this would solve my particular use case limiting the default time span between 2023-12-08T14:05:12.661Z and 2023-12-12T15:00:37.908Z?
(in my docker-compose as part of the EA service)

    command:
      - "--debug"
      - "--start"
      - "2023-12-08T14:05:12.661Z"
      - "--end"
      - "2023-12-12T15:00:37.908Z"

Furthermore, would this mean that this time frame will be the active/default time frame throughout the lifecycle of the EA instance for all rules registered in that particular instance?

I must applaud you for an extremely fast response - I'm not used to this kind of service in the "foss" world - simply top notch!!!! Thanks!

[jertel]

If I understand you correctly this would solve my particular use case limiting the default time span between 2023-12-08T14:05:12.661Z and 2023-12-12T15:00:37.908Z?

Since ElastAlert 2 is intended to be a long-running process, the end date actually needs to be in the future, otherwise it will immediately exit and log a message about the end date has already passed. But yes your start date can be as you have it.

Furthermore, would this mean that this time frame will be the active/default time frame throughout the lifecycle of the EA instance for all rules registered in that particular instance?

Yes, all rules would work against that time frame.

[litsegaard]

Thanks for your reply. I think I should've mentioned that we're running Praeco's ElastAlert docker container (together with Praeco) and, unfortunately, it doesn't support specifying --start and --end dates as parameters. Think I need to go back to the drawing board... Many thanks for your support though.

[nsano-rururu]

praeco is an unofficial tool. The elastalert2 docker image is officially released.
https://github.com/jertel/elastalert2?tab=readme-ov-file#docker-and-kubernetes