How do you alert on-call engineers about outages?
Grafana or Zabbix posts an alert to the on-call chat in Pachca, and its members get a notification. The on-call engineer reacts to the alert with 👀 so teammates can see someone’s already on it, then works through it in the thread. When the metric is back to normal, a recovery message arrives in the chat. Walk through the on-call engineer’s path yourself below.
7 members · Grafana bot
🔴 Firing · DiskSpaceLow
Less than 10% free space left on db-2
https://grafana.company.ru/d/db/postgres?viewPanel=7
✅ Resolved · DiskSpaceLow
Less than 10% free space left on db-2
https://grafana.company.ru/d/db/postgres?viewPanel=7
On call this week: , backup: . Whoever picks up an alert reacts with 👀 and posts what they’re doing in the thread. Once “Resolved” comes in, react to the alert with ✅
On-call
7 members · Grafana bot
🔴 Firing · DiskSpaceLow
Less than 10% free space left on db-2
https://grafana.company.ru/d/db/postgres?viewPanel=7
✅ Resolved · DiskSpaceLow
Less than 10% free space left on db-2
https://grafana.company.ru/d/db/postgres?viewPanel=7
On call this week: , backup: . Whoever picks up an alert reacts with 👀 and posts what they’re doing in the thread. Once “Resolved” comes in, react to the alert with ✅
Why get alerts in your team chat
Monitoring emails land in a shared inbox mixed in with everything else, and an alert in someone’s personal Telegram is seen only by the person who got it. The outage gets worked out in DMs and calls, and a week later nobody can piece together who rolled back what. In the on-call chat, everyone on shift sees the alert, and the triage stays in the thread under it.
- Everyone on call sees the alert. The bot posts to a chat, not to one person. If the on-call engineer is on the subway, their backup will spot the outage, and the team lead can see what’s happening in prod without a separate report.
- You can see right away who’s on it. 👀 on an alert means “I’m looking.” Two people don’t fix the same thing, and an alert with no reaction stands out right away.
- The chat stays a feed of alerts. Logs, hunches and “rolling back the release” go in the thread under the bot’s message, so each outage gets its own thread. You can @mention the service’s developer in the thread, even if they’re not in the chat.
- Recovery shows up in the same place. When the metric is back to normal, the monitoring system posts a message on its own. You don’t need to keep a dashboard open to know the outage is over.
- Only alerts come through at night. The on-call engineer adds the chat to their notification exceptions. All other work chats stay quiet until morning, but alerts still trigger a notification.
- A head start on the postmortem. The alert time, the first steps and the recovery time all sit in one thread. You can find it by searching for the alert name and pull it into the incident review.
How to set up alerts from Grafana, Zabbix and Prometheus
Grafana, Zabbix and Prometheus Alertmanager can send alerts by webhook, which is an HTTP request to a given URL. A bot in Pachca gives you that URL: whatever arrives there, the bot posts as a message in the chats it's been added to. When the metric goes back to normal, the monitoring system sends another request, and a recovery message shows up in the chat.
You can set up the bot and alert delivery in about 15 minutes, no developer needed. You'll need access to alert settings in your monitoring system and chat admin or creator rights in Pachca.
In Pachca
- 1
Create a chat for alerts
Click + next to search, choose Chat and name it "On-call." Add everyone in the on-call rotation, plus the team lead. If each team owns its own services, create a chat per team, so alerts go to the people who can fix them. - 2
Create a bot
In the Integrations menu, choose Bots, click Create bot and give it a name, like "Grafana." The bot will sign its alerts in the chat with this name. - 3
Turn on the incoming webhook
On the bot's page, open the Incoming webhook tab, click Turn on and copy the Webhook URL. Your monitoring system will send alerts to this URL. To learn how to format the alert text with a template, see our Help Center. - 4
Add the bot to the chat
Open the On-call chat settings, click Add members, go to the Integrations tab and choose the bot. It posts alerts in every chat it's been added to, so for a second team it's easier to create a separate bot.
In your monitoring system
- 5
Add the bot’s URL to your monitoring system
In Grafana, Zabbix or Prometheus Alertmanager, add a webhook for alerts and paste the bot's Webhook URL. Turn on recovery notifications too, so a message arrives in the chat when the metric is back to normal.
In your team
- 6
Agree on the rules
The on-call engineer reacts to an alert with 👀 when they pick it up and posts their progress in the thread. Once the recovery message arrives and the cause is clear, they react to the alert with ✅. If the outage is outside their area, they @mention the service's developer in the thread. Threads in Pachca are cross-chat threads, so you can bring in even someone who isn't in the chat. It's handy to post the rules and this week's on-call engineer as a message in the chat, like in the demo above. - 7
Set up notifications for on-call engineers
To get only alerts at night, the on-call engineer clicks their profile, opens Notification settings, turns off notifications for the whole app and adds the On-call chat to Exceptions. Chats on the exceptions list still send notifications while all other chats stay quiet. You can set up the exceptions list in the desktop app.
To show a Grafana chart under the alert, like in the demo, you need a separate link preview bot. A developer can set it up using our guide.
Troubleshooting
- The test passed, but the chat is empty. On the bot's Incoming webhook tab, click Show last received object. If the request is there, check that the bot has been added to the chat. If it isn't, check the URL in your monitoring system.
- The request is there, but no message arrived. Compare the fields in your template with the last received object: if the template returns an empty string, Pachca doesn't send the message.
- Outages come in, but recoveries don't. Check that recovery notifications are turned on in your monitoring system.