Service availability incident on March 27, 2026
Root cause analysis and timeline of the service outage on March 27, 2026.
Incident summary
What happened
On March 27, 2026, from 09:44 to 12:59 (UTC+3), our service was unavailable to users on all platforms: web, Android and iOS. Some users were briefly able to log in, but their sessions kept dropping.
No data was lost.
The database slipped into a degraded state: query response times shot up with no matching increase in load. Neither adding capacity nor reducing load helped, and the database couldn't recover on its own. The service was restored after a database restart.
We apologize for this incident. Below are the timeline, root cause analysis and the steps we're taking.
Timeline
All times are UTC+3.
- 09:44 – Monitoring detected a sharp rise in database response time at normal load. We started investigating and posted that the incident had begun in the customer chats where the first reports came in.
- ~09:50 – Autoscaling kicked in and added capacity. Things didn't improve.
- 09:54 – We notified users in our official channel.
- ~10:00 – We manually added more resources, which didn't help. Even simple queries were taking abnormally long.
- 10:19 – Serious degradation confirmed. We manually updated the incident on the status page.
- ~10:30 – It became clear the problem was in the database itself. We tried reducing load by partially restricting access to the service, with no effect.
- ~10:30–11:58 – We tried to stabilize the database while also considering a failover to a healthy database replica, but chose a restart as the more proven option.
- 11:58 – Database restart started.
- 12:59 – Restart complete. The service was restored.
Root cause chain
- Why did the service go down? The database stopped processing queries in a reasonable time.
- Why? The database slipped into a degraded state with no visible increase in load.
- Why didn't adding capacity and reducing load help? The problem wasn't a lack of resources but the database's internal state.
- Why didn't the database recover on its own? Our initial hypothesis was a potential bug at the DBMS level. The exact cause is under investigation.
What we're doing to prevent this from happening again
The exact cause of the database degradation is under investigation. But we're focusing on architectural changes that will reduce the impact and recovery time no matter what the cause turns out to be.
Reducing impact
- Database sharding. We're splitting the database into independent segments so a similar problem won't affect every user and the database can be restored faster.
Faster recovery
- Debugging replica failover. Failover itself works, but switching back to the primary database was still being debugged, which is why we didn't use this option during the incident. We're working through the full cycle and running drills.
Update, March 28, 2026: root cause identified
Our investigation found the most likely cause of the database degradation.
For several days, a monitoring service connection had been open to the production database in a mode that wraps every operation in a single transaction for the entire session. While that transaction stayed open, the database couldn't clean up the accumulated change history. This is standard DBMS behavior that protects data integrity.
When the session was restarted, the database started a massive cleanup of several days' worth of accumulated data. This internal operation consumed all available resources and blocked normal processing of user requests.
That explains what we saw during the incident: user load wasn't growing, and adding capacity or reducing external load didn't help, because the resources were being consumed by an internal database process.
Additional actions based on the investigation
- No long-lived service sessions on the production database. These sessions are no longer allowed in our workflow.
- Transaction time limits. We're setting a database-level timeout, so transactions open longer than a set threshold will be closed automatically.
- Alerts for long-running transactions. We're adding monitoring for transactions older than N minutes to catch situations like this before they affect the service.