What happens when a server fails

The first 30 seconds: the standby takes over

The sections below follow a server failure in the order things happen.

A high-availability database has two members on two servers in different sites, two of the three stars of its constellation: a primary that takes reads and writes, and a standby that receives every change. Every server runs HAProxy, which listens on your database's port and forwards connections to whichever member is the primary. Your hostname has one address per member's server, so a client can reach the database through either of them.

Patroni runs next to PostgreSQL in each member and decides which one is the primary. The primary holds a leader lock that it renews every 10 seconds and that expires after 30. The lock is kept in the region's Kubernetes API, which stores it in etcd, so it only changes hands when a majority of the three etcd voters agree. A primary that cannot renew its lock stops taking writes, so two members never both accept writes as the primary.

When a server fails, it usually does not refuse anything: it stops answering. Its member can no longer renew the lock, and when the lock expires the standby takes it and promotes itself. The standby is promoted within about 30 seconds of the failure. With the default synchronous replication, a commit only returned to your application after the standby had it on disk, so no acknowledged commit is lost. The other replication modes trade a little of that safety for faster commits.

HAProxy on every server asks each member once a second whether it is the primary. A member that fails two checks in a row counts as down, and the sessions that went to it are cut, so no client stays attached to a member that is no longer the primary. As soon as the new primary answers, both of your hostname's addresses route to it. Connections that were open to the old primary are closed; your application reconnects to the same hostname and reaches the new primary.

After two minutes: the server is suspect

Patroni and HAProxy handle the failover on their own, day or night. Our orchestrator acts on a slower clock. When Kubernetes has reported a server as not ready for two minutes, we mark it suspect:

• No new databases are placed on it.
• Every database with a member on it shows degraded redundancy on its page: it is running, but on one server.
• Our staff are paged.
• Its address is removed from the hostnames of the databases it served, including the -primary and -fallback names. Our DNS records have a 60-second TTL, so resolvers that respect it stop handing out the address within a minute.

A name never goes empty. A resolver that is told a name does not exist may cache that answer for much longer than 60 seconds, which would be worse than an address that does not answer. So we add the new address before we remove the old one, and when only one server is left, both the -primary and the -fallback name point at it.

Until the address is removed, a client that picks it waits for its connect timeout and then tries the next address, which is why the timeout matters (see below).

After ten minutes: the server is fenced

Most server outages are short: a reboot, a provider's network problem, a kernel that hangs and is restarted. Rebuilding a database copy from backup takes far longer than a reboot, so we do not start it at the first sign of trouble. When a server has been down for ten minutes, we fence it:

• Its fencing epoch is raised, so anything it believed about its members before is out of date.
• It is labelled as fenced, so nothing is scheduled on it.
• It gets a Kubernetes NoExecute taint, which tells the cluster that nothing may keep running there.

Fencing is what makes it safe to build a replacement elsewhere. A server that is cut off from us is not necessarily switched off: it may still hold an old copy of a database, or even run a member that last saw itself as primary. Once fenced, it cannot take part again until it has been cleaned (see when the server comes back).

There is one exception. If more than half of the data servers in a region look down at once, we do not fence any of them. That pattern is much more likely to be a network partition, where we have lost sight of healthy servers, than a mass hardware failure. Evicting every database on the far side of a partition would do more harm than the partition itself, so in that case people decide.

Rebuilding the lost copy

After fencing, every member that was on the failed server gets a new home. We do not keep a spare server waiting: the target is chosen at that moment, from what is free then.

• It is in the same region.
• It is never the server of the surviving member: two copies on one server would not be redundancy.
• We prefer a server in the failed server's site, so the database ends up spread over two sites as before, and otherwise a server in another site.

The new standby is restored from your database's encrypted backups in object storage, not copied from the surviving primary. Copying from the primary would load the one server that is serving your traffic at the moment it has no backup of its own. Restoring from object storage leaves its disk and network to your queries. The new member then replays the changes since the backup and starts streaming from the primary.

A failed server may have held many members, and they all need rebuilding. We rebuild at most three at a time per region, and only while the surviving servers' disk I/O is under 70%, so the rebuilds do not slow down the databases that are still serving. The order is fixed: single-node databases first, because they are down, then high-availability databases on the largest plans.

Redundancy goes back to ok once the new member is a synchronous standby streaming within 1 MB of the primary. If no server in the region has room, the database stays degraded, shows that it is waiting for capacity, and we are paged to add a server; the rebuild starts as soon as one joins.

The primary does not switch back afterwards. Roles float: the member that was promoted stays the primary, and the rebuilt one becomes its standby. A switch back would cost you another round of reconnects for no gain, since both members are equivalent.

When the server comes back

A fenced server is not trusted when it starts answering again. Its disks may still hold volumes of databases that now live elsewhere, and a member on it may still believe it is a primary. It only takes databases again after three things:

1. Its cleaner has deleted every stale volume on it, so no old copy of a database can be served or read.
2. It has acknowledged the new fencing epoch, which shows that it knows it was fenced.
3. It has been ready for ten minutes, so a server that keeps crashing does not come back in between.

Then it is an ordinary server again and new databases can be placed on it. Nothing moves back to it: the databases that were rebuilt elsewhere stay where they are.

What to do in your application

The steps above keep your database available. Whether your application notices more than a short pause depends on how it connects.

• Set a connect timeout of 5 to 10 seconds. A failed server stops answering rather than refusing. Without a timeout, a new connection to its address waits until the operating system gives up, about two minutes on Linux.
• Use a driver that tries every address of a name. libpq (and so psql, psycopg, Ruby pg and PHP), pgx, asyncpg and Npgsql do: after the timeout on the failed address they try the other one.
• Use the multi-host connection string with drivers that use only the first address of a name, such as JDBC. It lists the -primary name, the -fallback name and the main hostname, and the driver tries them in order:

postgresql://u_<id>@<id>-primary.db.electradb.com:<port>,<id>-fallback.db.electradb.com:<port>,<id>.db.electradb.com:<port>/app?sslmode=verify-full

The database page shows this string with your own names and port. Add your driver's connect timeout to it: each name is tried for that long before the next.

• Reconnect and retry on connection errors. Connections to the old primary are closed during a failover. Use a driver or connection pool that reconnects on errors, and run a transaction that was cut off again.

The settings for each driver, and why your servers should use a resolver that keeps our 60-second TTL, are in the Quickstart under timeouts and failover.

Single-node and Dev plans

A single-node database has one member, so there is no standby to promote. When its server fails, the database is down. It is restored from backup on another server in the same region when the failed server is fenced, ahead of every high-availability rebuild, and it keeps its hostname and connection string: new connections reach it as soon as it is back.

The restore replays everything that was archived. PostgreSQL archives its changes at least every 60 seconds, so writes from up to about the last minute before the failure, not yet archived, can be lost. For your application, the connect timeout and reconnecting on errors are all you need.