How to avoid flathunter.exceptions.StampedeProtectionException

[conorheins]

First of all, thank you very much for creating this great app!

I ran into an issue when getting my first set of results from flathunter, seemingly due to the the StampedeProtectionException. Can anyone suggest a way to change the config file to avoid this?

It seems like too many messages are being received too fast, so the app craps out and terminates itself. See screenshot for detailed error message below:

With the key lines seeming to be this trace back:

Traceback (most recent call last):
  File "flathunt.py", line 118, in <module>
    main()
  File "flathunt.py", line 114, in main
    launch_flat_hunt(config, heartbeat)
  File "flathunt.py", line 36, in launch_flat_hunt
    hunter.hunt_flats()
  File "/Users/conor/flathunter/flathunter/hunter.py", line 56, in hunt_flats
    for expose in processor_chain.process(self.crawl_for_exposes(max_pages)):
  File "/Users/conor/flathunter/flathunter/sender_telegram.py", line 37, in process_expose
    self.__broadcast(
  File "/Users/conor/flathunter/flathunter/sender_telegram.py", line 61, in __broadcast
    self.__send_images(chat_id=receiver, msg=msg, images=images)
  File "/Users/conor/flathunter/flathunter/sender_telegram.py", line 124, in __send_images
    self.__handle_error(
  File "/Users/conor/flathunter/flathunter/sender_telegram.py", line 159, in __handle_error
    raise StampedeProtectionException(
flathunter.exceptions.StampedeProtectionException: Too many messages too fast - backoff 9 seconds

In case relevant, I also leave my config yaml below.

loop:
    active: yes
    sleeping_time: 600
urls: 
    - https://www.immobilienscout24.de/Suche/...
filters:
message: |
    {title}
    Zimmer: {rooms}
    Größe: {size}
    Preis: {price}
    Ort: {address}

    {url}
captcha:
      imagetyperz:
            token: [REDACTED]
      driver_arguments:
            - "--no-sandbox"
            - "--headless"
            - "--disable-gpu"
            - "--remote-debugging-port=9222"
            - "--disable-dev-shm-usage"
            - "window-size=1024,768"
notifiers:
    - telegram
telegram:
    bot_token: [REDACTED]
    notify_with_images: true
    receiver_ids:
        - [REDACTED]

Any help in figuring this out would be really helpful -- thanks in advance :)

[codders]

Hi @conorheins ,

When you first run the app, it will find a lot of listings and send a lot of messages. This triggers the HTTP 429 error from the Telegram messaging API.

The StampedeProtection is supposed to then pause (backoff 9 seconds) before sending more messages. Does the app actually crap itself and terminate at that point? Or does it sit there until the next loop?

Usually after the first couple of runs, the local database (processed_ids.db) has cached most of the ads, and the message sending happens at a rate that Telegram can cope with.

[conorheins]

I see, that makes sense. Yeah unfortunately in my case the app actually just terminated entirely, so the Python program was actually interrupted. So it wasn't idling/waiting for the next loop unfortunately.

[codders]

Looking at the code (https://github.com/flathunters/flathunter/blob/main/flathunter/sender_telegram.py#L154-L161), it seems that we just raise the exception and never handle it. But the sleep is already in there - it might be enough just to swallow the exception (or not even raise it in the first place). I'll see if I find time to mess with this this week (but of course, pull requests welcome if you have a setup where you can reproduce the issue! :) )

[conorheins]

Good to know, thanks -- I will try to just comment that block of lines out / not raise the exception, if i get to a place where I can reproduce the issue. My current problem is now that IS24 is constantly detecting me as a bot (aka our discussion in #302), so I'm not even getting that far.