Postmortem
Service availability incident on March 6, 2026
Root cause analysis and timeline of the service outage on March 6, 2026.
Incident summary
Date and timeMar 6, 2026, 13:28–13:32 (UTC+3)
CauseA mass drop of WebSocket connections set off a flood of reconnects that overloaded the service
SeverityCritical
Duration~35 seconds of full downtime + a tail of degraded performance until ~13:32
Affected systemsWeb, Android, iOS: all platforms
Affected users716 unique users saw errors
What happened
On March 6, 2026, at 13:28 Moscow time, Pachca's web clients all lost their WebSocket connections at once. With so many clients reconnecting at the same moment, the load spiked and briefly exceeded what our infrastructure could handle, so users saw errors. The system recovered on its own after ~35 seconds.
No data was lost.
We apologize for this incident. Below are the timeline, root cause analysis and the steps we're taking.
Timeline
All times are UTC+3.
- ~13:27 – WebSocket connections drop en masse.
- 13:28 – A flood of reconnects begins. Load on the load balancer spikes.
- 13:28:38 – First API errors: the backend can't keep up with the volume of requests.
- 13:29 – Peak load: the load balancer is overloaded, and some requests never reach the backend.
- 13:29:13 – Last errors: the API starts to recover.
- 13:30 – Load on the load balancer returns to normal.
- ~13:32 – The system is fully recovered.
Root cause chain
- Why did the service go down? The infrastructure couldn't handle a flood of simultaneous requests.
- What caused the flood? All web clients lost their WebSocket connections at the same time and started reconnecting.
- Why did reconnecting overload the system? So many clients reconnecting at once put a brief but critical load on the API.
- What caused the initial connection drop? We've identified the cause and are running additional analysis.
What went well
- The system recovered on its own, with no manual intervention.
- Most mobile clients stayed connected.
- Monitoring captured all the metrics we needed for the retrospective.
- Logs preserved a detailed picture of the errors.
What went wrong
- The simultaneous reconnect pattern created load beyond normal operating limits.
- Monitoring didn't catch the anomaly early on.
What we're doing to prevent this from happening again
- Smarter reconnects. We're improving how clients behave when they reconnect to reduce peak load.
- Better observability. We're strengthening monitoring to catch anomalies earlier.
- More resilient infrastructure. We're adding infrastructure-level protection against load spikes.
Lessons learned
This incident showed that a large number of clients reconnecting at once can create load comparable to an external attack. We're making the system more resilient to scenarios like this.