Яндекс.Метрика
Postmortem

Service availability incident on November 17, 2026

Timeline of the service availability incident on November 17, 2026. The issue was resolved and normal service performance was restored at 2:43 p.m.

Incident summary

Date and timeNovember 17, 2026 Started: 12:39 p.m. (MSK)
CauseRedis server failures at the cloud provider and exhausted memory, with no automatic failover to the standby node
SeverityCritical
StatusIssue resolved. Normal service performance was restored at 2:43 p.m.
DurationFull outage: 16 minutes (12:39–12:55 p.m.). Slower loading and operations: 65 minutes in total (1:01–1:13 p.m. and 1:50–2:43 p.m.)
Affected systemsWeb, Android, iOS: all platforms
Affected usersAll users during the full outage. During the slowdown, the impact depended on the actions users were performing
Last updatedNovember 17, 2026, 2:43 p.m. (UTC+3): issue resolved and normal service performance restored

What happened

On November 17, 2026, Pachca experienced service disruptions: the service was unavailable for 16 minutes. It also ran slowly for a total of 65 minutes: loading and operations took longer, and some operations failed, including saving drafts.

The disruptions were related to Redis hosted by our cloud provider. Repeated server failures did not trigger an automatic switch to the standby node: between failures, the server resumed sending availability signals. Redis also ran out of memory, and its configuration prevented it from automatically removing existing data to free up space, causing write errors.

Normal service has been restored. We apologize for this incident. Below are the timeline, root cause analysis and steps to prevent similar failures.


Timeline

Event time zone: (UTC+03:00) Moscow, Saint Petersburg.

  • 12:39 p.m. — Monitoring detected a sharp increase in service response time. The first support requests arrived. We began investigating and working to restore service. We posted incident updates in the customer chats where the reports came in.
  • 12:55 p.m. — The problem was localized. Service availability and normal performance were restored.
  • 1:01 p.m. — Loading and operations slowed down again.
  • 1:13 p.m. — Normal loading and operation speeds were restored.
  • 1:50 p.m. — Redis availability issues recurred, and loading and operations slowed down again.
  • 2:43 p.m. — The issue was resolved. Normal service performance was restored.

Root cause chain

  1. Why was the service disrupted? Redis experienced availability issues. The cloud provider reported repeated server failures; exhausted memory was also identified.
  2. Why did users see draft-saving errors and unavailability messages while the service was still partly working? Our preliminary assessment is that Redis responded intermittently but could not write new data once its memory was full: the noeviction setting prevented existing data from being removed to free up space. Partial service availability therefore did not mean that every operation, including saving drafts, could succeed.
  3. Why did Redis not automatically switch to the standby node? According to the cloud provider's team, after each failure the server recovered and resumed sending availability signals. It resumed sending them before the system declared it unavailable and triggered failover. As a result, repeated failures did not lead to a switch to the standby node.

What we're doing to prevent this from happening again

We have made changes to keep the core service running if a similar failure recurs. We are also exploring ways to reduce Redis recovery time.

  • Reducing impact

    Protected the core service against failures of this Redis instance. Another failure of this instance will not slow down the API.

    Improved processing of queued tasks. A backlog of tasks no longer affects the core service.

  • Faster recovery

    Exploring failover to the standby node during repeated brief failures. The Pachca team will examine the scenario where the server sends availability signals between failures but remains unstable, preventing automatic failover. A solution has not yet been selected.