Long Alerts and Message Fatigue: A Prometheus Story

I've been getting screamed at by my lab for a few years now. What started in the pursuit of a sysadmin-ish secondment opportunity somehow turned into constant Discord alert messages and, even more disappointingly, still getting Service Desk calls.

But damn it all, if I get the opportunity to learn about a thing, you bet your bottom dollar I’m going to try it, break it, curse at it and then rebuild it (rinse and repeat) in my own little rack that could. So Prometheus, Blackbox, Alertmanager and Grafana it was!

From 2022 until early 2026, there it sat: half complete and still screaming, while I focused on pretty graphs to grace the top of the rack. It DID look pretty though 😍

So for that entire time, the alerts were always long and constant. I don’t know why I never treated that as a problem but I guess I was getting alerted, right?

After the K3S pivot from multiple VM's all running 3-5 docker containers (why past Mitch, why. Just put it on the same host damn you) I finally fixed it, and boy was it easy. I genuinely don’t know how I missed it earlier, but for those playing along at home, this is what I cooked up:

{{- define "discord.slim.title" -}}
{{- if eq .Status "firing" -}}
🔥 [FIRING:{{ len .Alerts.Firing }}] {{ .CommonLabels.alertname }}{{ with .CommonLabels.severity }} ({{ . }}){{ end }}
{{- else -}}
✅ [RESOLVED] {{ .CommonLabels.alertname }}
{{- end -}}
{{- end }}

{{- define "discord.slim.message" -}}
{{- range .Alerts }}
📍 **Cluster:** `{{ .Labels.cluster }}`
🎯 **Job:** `{{ .Labels.job }}`
{{- with .Labels.instance }}
🖥️ **Instance:** `{{ . }}`
{{- end }}
{{- with .Labels.namespace }}
📦 **Namespace:** `{{ . }}`
{{- end }}
{{- with .Annotations.summary }}
📝 **Summary:** {{ . }}
{{- end }}
{{- with .Annotations.description }}
📖 **Details:** {{ . }}
{{- end }}
{{- with .Annotations.runbook_url }}
🔗 **Runbook:** <{{ . }}>
{{- end }}

{{- end }}
{{- end }}

(Note: Can't say I fully came up with this on my own. It became an amalgamation of stuff that I learnt, other various sites and AI telling me what I did wrong to have it NOT work)

Save the template as something like custom.tmpl, then make sure Alertmanager loads it via the templates: section. In my case, the template is mounted inside the Alertmanager container under /etc/alertmanager/config/, so my config uses:

templates:
  - '/etc/alertmanager/config/*.tmpl'

and your receiver would look a little something like this:

receivers:
      - name: discord
        discord_configs:
          - webhook_url: "WOWLOOKAWEBHOOKADDRESS"
            title: '{{ template "discord.slim.title" . }}'
            message: '{{ template "discord.slim.message" . }}'

If you are running Alertmanager directly, this path needs to match where the file exists on disk. If you are running it in Kubernetes/Helm, the template path needs to match where the ConfigMap or Secret is mounted inside the container. Hope you remembered how you set that up 3 months later because I, dearest reader, did not. The hindsight version of this is just use Docker for your alert setup if you can. Your head hurts less 😅

Then tune your alert for: duration and Alertmanager timing values so it only yells when something is actually worth yelling about. Hey presto, no more spamerino.

Scientists are still investigating why I put up with it until now, but hey, it’s all pretty now 🤦‍♂️