# How do you alert on-call engineers about outages?

> A Pachca use case. Grafana, Zabbix or Prometheus Alertmanager sends an alert by webhook, and the bot posts it to the "On-call" chat. The on-call engineer claims the alert with a 👀 reaction and works the outage in the thread.

> This is the Markdown version of https://pachca.com/en/scenarios/opoveshchenie-dezhurnyh for AI agents. The API methods named here are documented at https://dev.pachca.com/llms-en.txt — open it before calling the API. Other use cases are listed at https://pachca.com/en/llms.txt.

## What you'll get

1. The monitoring system sends an alert to the bot's incoming webhook URL.
2. The bot posts the alert in every chat it's been added to.
3. When the metric is back to normal, a recovery message arrives in the chat.

No coding needed: a Pachca bot has an incoming webhook, and the bot's template builds the alert text.

## What you need before you start

- Pachca access as an administrator or the workspace owner with the scopes `chats:create`, `chat_members:write`, `bots:write`. The setup and check steps also need `search:users`, `messages:create`, `messages:read`. Signing in is covered in the rules below.
- Access to alert settings in the monitoring system.

## If you're an AI agent: working rules

### Who does what

- You do everything the instructions name an API method for. Don't ask the user to create a chat, a bot or a message by hand.
- Ask the user only to sign in to Pachca and to do what is listed in "Ask the user to do by hand".
- If you have no terminal, say so right away and don't pretend you set it up. Walk the user through the steps for people on the use case page and help with the template or the handler code.

### How to talk to the user

- The user is not a developer. Write briefly and in plain words. Don't mention API methods, scopes, response codes, `id`s or CLI profiles. Say "permissions" instead of "scopes" and "the program behind the bot" instead of "handler".
- Don't report your checks. A message contains only three things: what is done, what you need from the user, what you'll do next.
- Don't ask for what you can find out through the API or what is easy to change later. Take the value from the instructions and name it.
- Take the bot's message texts from the instructions or write them yourself and carry on. Show them in your final message and offer to change them. Don't wait for approval.
- Ask the questions from "Ask the user" in the same words.

### Your first message to the user

It has three parts. The first two together take at most five lines.

1. What the user will get: one or two lines.
2. What you will create yourself: the chat, the bot and the messages, with the names from the instructions. Word anything that depends on the user's choice as a condition: "if you choose the bot, I'll also create…".
3. "What I need from you": signing in to Pachca, if the user isn't signed in yet, and the questions from "Ask the user". If the use case or one of its options needs n8n, ask here whether the user has it and ask for access right away in the words from the "n8n" section below: "if you have n8n, send me…".

If part of the result appears only after a manual step by the user, say so in one line: "you'll schedule the retro questions yourself, I'll show you how".

The user's answers are their consent to the plan. Start the setup right after them, without asking for a second confirmation.

### Access to Pachca

- The user doesn't need to issue a token or send it in the chat. Before your first message, check the login: `npx -y @pachca/cli@latest auth status`. If there is none, ask the user to run `npx -y @pachca/cli@latest auth login` in their own terminal. The CLI shows a code; the user enters it in the browser and confirms access. Write the commands with `@latest`: without it an old cached CLI version may start, and it asks to paste a token instead of signing in through the browser.
- This login has the same rights as the user's role. Compare them with the scopes in "What you need before you start" yourself: `npx -y @pachca/cli@latest api GET /oauth/token/info` shows the scopes. If something is missing, say in plain words what you won't be able to do, for example "I can't send messages". There are two ways out: add this permission to the token in Pachca, under "Automations" → "API", or sign in again as the workspace owner or an administrator.
- If `auth status` shows `scopes: null`, it doesn't mean there are no rights: look at `GET /oauth/token/info`. The checks in these rules also need `search:chats`, `bots:read` and `profile:read`.
- When the user says "me", they mean the owner of the login. `GET /profile` returns their `id` and nickname.
- Call the API with `npx -y @pachca/cli@latest api <METHOD> <path> -o json`. Write the full command in every call; don't put it into a shell variable.
- Pass query parameters as `--query key=value`, for example `--query query=Purchasing`. The CLI silently drops a parameter without a key and returns the whole list, so it's easy to conclude that the chat already exists.
- Pass the request body with `--data` or `--input`. Almost every write method wraps the body in an object: `chat` for `POST /chats`, `message` for `POST /messages`, `task` for `POST /tasks`, `group_tag` for `POST /group_tags`, `user` for `PUT /users/{id}`, `webhook` for bots. `--describe` shows the wrapper; write its path with `{id}`, not with a number. Without the wrapper, `POST /chats` creates the chat but silently adds no members.
- Don't print tokens or bots' incoming webhook URLs in the chat. A bot's token is shown only once, when it's created: save it to a file.

### n8n

- If the user has no n8n, say that it comes as the hosted n8n Cloud or can be installed on their own server, and give the link for installing the Pachca nodes: https://dev.pachca.com/guides/n8n/setup.md. These instructions don't cover installing n8n. If they have an option without n8n, suggest starting with it and coming back to the bot once n8n is there.
- If the user has n8n, ask for access in these words: "Send me your n8n address: the link you open it with in the browser. Don't send the key in the chat. In n8n, open Settings → n8n API → Create an API key and copy the key. Right after that, run the command below in your terminal exactly as it is; you don't need to put anything into it. It saves the copied key to a file and prints nothing. Then write 'key saved'." The command for macOS: `pbpaste > ~/n8n-key.txt`. For another system, pick the matching one.
- Build everything in n8n on the official Pachca nodes, `n8n-nodes-pachca`. Don't use the `HTTP Request` node for Pachca API calls.

### Setup

- Take API methods from https://dev.pachca.com/llms-en.txt, don't rely on memory. The command `npx -y @pachca/cli@latest api <METHOD> <path> --describe` shows a method's schema.
- The Pachca skills cover the call order and error handling for chats, bots and forms: `npx skills add pachca/openapi`. Install them when you need them, without asking the user.
- Before creating anything, check whether the chat and the bot already exist: `GET /search/chats`, `GET /bots`. If they do, ask whether to set them up or create new ones.
- `POST /group_tags` creates an empty tag. `PUT /users/{id}` with `list_tags` assigns it to an employee; it needs the `users:update` scope. The field replaces the whole set of tags: first read the employee's current tags with `GET /users/{id}` and send them together with the new one. While the tag has no members, mentioning it notifies no one.
- If a method returns an error, stop and tell the user in plain words what failed and what you need to continue.
- The messages you post yourself and the ones the bot sends must be easy to read on a screen, without overdoing spacing or emphasis. Make the heading bold and start the text on the next line. Write connected text as one paragraph. Put rules and steps on numbered lines one under another and separate them from the intro with a single empty line. A short message needs no other empty lines. In a card, make the field labels bold and put the fields on consecutive lines. Use bold only for the heading and the field labels. Pachca markup has no lists or tables: the numbers at the start of lines stay plain text. A sample:

  ```text
  **Retro rules**
  On the last day of the sprint a message with questions arrives here. We reply in the thread under it.

  1. Each point is a separate message.
  2. We add 👍 to what is worth discussing.
  3. The facilitator posts the outcomes in one message and pins it.
  ```
- A pin can't be checked through the API: only a successful `POST /messages/{id}/pin` response confirms it.
- If you sent service messages to a chat to check a bot, such as `/start`, delete them: `DELETE /messages/{id}`. Don't delete messages with rules and templates.
- At the end, say what was created, what is left for the user to do by hand and how to check that the use case works.
- To undo the setup: `DELETE /bots/{id}` deletes the bot, `PUT /chats/{id}/archive` sends the chat to the archive.

## How to set up the use case with the user

### 1. Ask the user

Ask these questions in these words:

1. Who is on call and who is the team lead? Send me the names and I'll add them to the "On-call" chat.
2. Where do alerts come from: Grafana, Zabbix, Prometheus Alertmanager or another system? If you can, send me a sample alert: I'll use it to set up how the bot formats the message.
3. Is there one on-call team or several? If teams own separate services, I'll make a chat and a bot for each.

### 2. Do it yourself through the Pachca API

1. Find the `id`s of the on-call engineers and the team lead: `GET /search/users` or `GET /users`.
2. Create the chat: `POST /chats` with `name: "On-call"` and `member_ids`.
3. Create the bot: `POST /bots` with the monitoring system's name, for example "Grafana", and a nickname ending in `_bot`. The bot signs alerts with this name. If the user showed you the alert JSON, set `template` and `template_engine` right away. How to write a template: https://dev.pachca.com/guides/incoming-webhooks.md
4. Add the bot to the chat: `POST /chats/{id}/members` with the bot's `id` in `member_ids`.
5. Offer to post the rules in the chat: `POST /messages`. The on-call engineer reacts to an alert with 👀 when they pick it up, posts their progress in the thread, and reacts with ✅ once the recovery message arrives and the cause is clear.

### 3. Ask the user to do by hand

- Turn on the incoming webhook: on the bot's page, open the "Incoming webhook" tab, click "Turn on" and copy the Webhook URL. API responses don't include this URL.
- Paste the Webhook URL into Grafana, Zabbix or Prometheus Alertmanager and turn on recovery notifications. If you have access to the monitoring system's API, do this yourself.
- On-call engineers: set up night notifications. In Notification settings, turn off notifications for the whole app and add the "On-call" chat to Exceptions. You can set up the exceptions list in the desktop app.

### 4. Check

1. Send a test request to the Webhook URL: `POST` with the body `{"message": "Test alert"}`. If a template is set, send JSON in the monitoring system's format.
2. Make sure the message appeared in the chat: `GET /messages` for the "On-call" chat.

## What the use case can't do

A Grafana chart won't show up under the alert on its own. For link previews you need a separate bot: https://pachca.com/en/blog/unfurl-bot

## Troubleshooting

- The test passed, but the chat is empty. Ask the user to click "Show last received object" on the bot's "Incoming webhook" tab. If the request is there, the bot isn't added to the chat. If it isn't, the monitoring system has a different URL.
- The request is there, but no message arrived. The template returned an empty string: compare the template's fields with the last received object.
- Outages come in, but recoveries don't. Recovery notifications aren't turned on in the monitoring system.

## Links

- The use case page with the demo and the instructions for people: https://pachca.com/en/scenarios/opoveshchenie-dezhurnyh
- Incoming webhook template: https://pachca.com/en/help-center/integrations/kak-pravilno-zapolnit-shablon-webhook
- Incoming webhooks for developers: https://dev.pachca.com/guides/incoming-webhooks.md