Alerts works only manualy

[NoFWayMan]

Hi! I used a docker container with bolt storage, and when i start alertmanager-bot container - it sends me alerts, that allready in "firing" status at prometheus "Alerts" page.
But after that bot stops sending me alerts automaticly, only if i type /alerts in Telegram chat.
What can be the problem?

[metalmatze]

How often is Prometheus / Alertmanager configured to send alerts?

[NoFWayMan]

30 seconds for "resolve_timeout" in alertmanager.conf

global:
  resolve_timeout: 30s
route:
  group_by: ['alertname', 'instance', 'severity']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'alertmananger-bot'
receivers:
- name: 'alertmananger-bot'
  webhook_configs:
  - send_resolved: true
    url: 'http://alertmanager-bot:8080

1 minute in particular rule, that i want to trigger in prometheus/rules

  - alert: CriticalDiskSpace
    expr: 100 - ((node_filesystem_avail_bytes{alias="hadoop_nodes",mountpoint="/bdp"} * 100) / node_filesystem_size_bytes{alias="hadoop_nodes",mountpoint="/bdp"}) > 80
    for: 1m
    labels:
      severity: critical
    annotations:
      description: "{{ $labels.instance }} is more than {{ $value }}% used."
      summary: "Instance {{ $labels.instance }} is low on disk space"

[metalmatze]

@NoFWayMan repeat_interval is set to 1h so it'll only resend the same notifications every hour.

[NoFWayMan]

Got it. Set it to 1m for testing - still not getting triggered alerts
I restarted prometheus and alertmanager to be sure, re-build container. Still working only on /alerts request only :(

[metalmatze]

@NoFWayMan Is the http://alertmanager-bot:8080 addressable by alertmanager? Do you see anything in the logs of alertmanager pointing to that failing? Do you see the alerts in alertmanager to beginning with?

[NoFWayMan]

http://alertmanager-bot:8080 is just an example URL in this situation. There is an actual URL of host.
At the /api/v1/alerts of alertnamager i see the alert. Also i have a record in bot.db, that contains my Telegram data (username, chat id, etc).
No error logs from alertmanager. It seems like everything working just fine, but at some reason msgs didn't get to Telegram.

[KesleyDavid]

Same problem for me.

If I use the example curl the notification will also arrive on my telegram.

Apparently it’s the alertmanager who’s not calling the bot endpoint