Postmortem
Service availability incident on August 18, 2025
Root cause analysis and timeline of the service performance degradation on August 18, 2025.
Incident summary
Date and timeAug 18, 2025, 10:53 (UTC+3)
CauseAn update that optimized @mentions, released on Aug 11, 2025, put heavy load on an old query
SeverityHigh
Duration43 minutes
Affected systemsapp.pachca.com: web, mobile and desktop apps
Affected usersAll users
What happened
On August 18, 2025, at 10:53 Moscow time, Pachca started slowing down. The cause was an update that optimized @mentions, released a week earlier (Aug 11, 2025), which put heavy load on an old query. The slowdown lasted 43 minutes.
No data was lost.
We apologize for this incident. Below are the timeline, root cause analysis and the steps we're taking.
Timeline
All times are UTC+3.
- 10:53 – We started getting a large number of reports that the service was slow.
- 10:55 – The team started investigating and narrowing down the problem.
- 11:36 – Full access to the service was restored.
Root cause chain
- Why did the service slow down? An old database query was hit with abnormally high load.
- Why did the load increase? The Aug 11, 2025 update that optimized @mentions changed the request pattern, which overloaded the old query.
- Why didn't the problem show up right away? Load built up gradually over the week after the update and hit a critical level on Aug 18, 2025.
What we're doing to prevent this from happening again
Immediate actions
- Temporary limits on @mentions. We limited @mentions to take load off the system and restore the service.
Long-term actions
- Fixing @mention search. We're fixing issues with searching for @mentions in chats and threads.
- Incident response protocol. We're introducing a more structured response protocol to speed up diagnosis and recovery.
- Performance optimization. We're continuing to work on the app's performance and stability.