Geo-redundancy and fault tolerance at Pachca
Customers ask what happens to Pachca if a data center goes down. Here is how our infrastructure is set up and what happens when a site fails.
Where Pachca runs
Pachca runs in seven Selectel data centers in Moscow and Saint Petersburg. Every site is in Russia.
Most requests are handled in the Moscow ru-6 region: three data centers in different locations that work as one. If one fails, traffic moves to the other two. Selectel classifies this region as disaster-tolerant.
Our entry point, api.pachca.com, runs on Anycast, so network traffic is spread between Moscow and Saint Petersburg. If one city is unreachable, requests go to the other.
What happens when a data center fails
Every critical Pachca component is duplicated, so the service survives the loss of a single site by failing over to the others.
The primary database has two active replicas ready to take the load. Most services fail over automatically using MaxScale and Redis Sentinel. A few services are switched manually by on-call engineers, who are paged around the clock.
The three ru-6 sites share the load between them.
How we store backups
We take full and incremental backups of the database and keep the transaction log in between, so nothing is lost between snapshots. Backups are encrypted and shipped to several independent sites. Every few days they are also moved to cold storage in different regions.
- 01Full backupA snapshot of the whole database
- 02IncrementalOnly what changed since the last snapshot
- 03Transaction logSo nothing is lost between backups
- 04Independent sitesEncrypted and shipped to several sites
- 05Cold storageEvery few days, in different regions
What's next
We've been building failure-tolerant infrastructure for years and keep expanding redundancy.