Constellations: the three servers behind every database

Stars and constellations

Picture every server we run in a region as a star in the sky. Each one sits in a site, a data centre of its own, with its own disks, and is joined to the others by a private network. One star on its own cannot keep a database available: if it goes out, whatever it held goes with it.

A constellation is the shape your database draws between those stars: the three servers it relies on. Two of them hold its copies, the primary and the standby, in different sites. The third holds the deciding vote when the question is which of the two is the primary. Different databases draw different constellations from the same sky, and a constellation can change shape during its life: when a server fails, its star is replaced by another one.

A single-node database is a single star: one server holds it, and if that server fails, the database is restored onto another.

All the stars of a region belong to one Kubernetes cluster (k3s), which our code and runbooks call a cell, and a constellation never reaches outside its region: failover, recovery and plan changes all happen among the stars of the region you chose. The control plane described in Inside ElectraDB is not a star in any constellation. It decides which stars a database uses, but it holds no copy and no vote, and everything below works without it.

Three votes decide who is primary

The three stars of a constellation hold the votes. They run the etcd database that k3s embeds, and etcd only accepts a change when a majority agrees: two of three. Patroni, which runs every PostgreSQL cluster, keeps its leader record in Kubernetes, and so in that etcd. A member is the primary only while it holds that record and can renew it.

This is what the third vote buys:

• No split brain. Two servers on their own cannot tell a dead partner from a broken link between them, and both could decide to be primary. With a third vote, the side that keeps the majority decides. A primary cut off from it cannot renew its leader record and stops accepting writes.
• No third copy of your data. The third star holds a vote, not a copy: it runs no PostgreSQL for your database. A high-availability database needs two copies, and you pay for two.
• Constellations share their voting star. Many databases rely on the same quorum, which we monitor and look after once, instead of one voting setup per database.

Losing one voting server costs nothing: two of three is still a majority, and every database keeps its primary or promotes its standby as usual.

Local disks, a private mesh and a fence around each database

Your data sits on fast local disks (NVMe or SSD) in the server that runs each member, on XFS with a quota per volume, so one database can never fill a disk another database needs. There is no network storage. Copying your data to a second site is the job of PostgreSQL's own replication, managed by Patroni, which knows what a committed transaction is. A storage system underneath would add a second layer of replication and a second place for things to go wrong.

The servers talk to each other over a WireGuard mesh, so replication and cluster traffic between sites is encrypted and never leaves that private network. Round trips between the sites are 11 to 15 milliseconds. That is the price a synchronous commit pays to have your change on disk in a second site before it returns (see High availability and replication).

On every server, Cilium enforces network policy for every database separately. Each database lives in its own Kubernetes namespace that refuses all traffic by default. Its members accept connections only from the routing layer, from each other and from monitoring, and they can reach only what they need, such as the backup storage.

Every server routes to the primary

Every data server runs HAProxy, listening on the port of every database in the region. Once a second, it asks each member of each database whether it is the primary, using Patroni's /primary check, and sends new connections to the one that says yes. When the primary changes, HAProxy follows within seconds. It does not read or end TLS: the encrypted connection passes straight through to PostgreSQL, which presents the certificate for your hostname.

Your hostname has one address for each server that holds a member of your database, with a 60-second TTL. Whichever address your driver tries, that server routes you to the primary, so DNS never has to pick the primary itself. High-availability databases also get two names with one address each: -primary follows the server running the primary, and -fallback points at the other. They exist for drivers such as JDBC that try only the first address of a name; the multi-host connection string on the database page lists both and the main hostname.

How a constellation is drawn

When you create a high-availability database, we draw its constellation: two servers in different sites, of those that fit the two with the most free memory, joined by the server that holds the deciding vote. A single-node database gets the one server with the most free memory. Memory, disk and the dedicated vCPU of paid plans are reserved for your database and never overcommitted, so a busy neighbour cannot take what your plan includes. If no servers have room, the create is refused straight away with a clear message, before anything is built or billed.

After that, nothing moves your database for the sake of tidiness. Roles float: if a failover makes the standby the primary, it stays the primary. Moving it back would mean a second interruption for no gain, because both servers are the same. Members move only when a server fails, when you change plan or when we retire a server.

A database also never leaves its region. A synchronous standby on another continent would add 80 to 250 milliseconds to every commit, so both copies and the backups stay in the region you chose (see Regions and data residency).

When a server fails

A constellation repairs itself when one of its stars goes out, in this order:

1. Patroni promotes the standby on the surviving server, usually within about 30 seconds, and HAProxy on every server starts sending connections to it.
2. About two minutes after the server stops answering, it is marked as suspect, its address leaves your hostname and no new databases are placed on it.
3. If it stays down for ten minutes, it is fenced: it may not serve again until it has deleted every copy of data it no longer owns. Your database gets a new standby on another star, preferably in the failed server's site, so the constellation keeps its shape. The standby is rebuilt from backups rather than from the surviving primary, so the server still serving you is not slowed down by the copy.

A single-node database has no standby, so it is down until it is restored from backup onto another server, with the same hostname. Those restores go first. The high-availability guide describes what your application sees.

Monitoring from outside

Each region collects metrics from its servers, HAProxy, Patroni and every database, and pushes them over the mesh to every control server. The control servers keep the history, evaluate the alert rules and page us. Losing a server costs about a minute of its metrics, not the monitoring. If a whole region stops sending, that silence is itself an alert, which monitoring that lived only among the region's own servers could never raise.