Inside ElectraDB: how the service is put together

The parts

ElectraDB is a small number of parts with one job each. Knowing them makes it easier to judge what happens to your database when something goes wrong, and which failures can reach it at all.

• The website: a static application that runs in your browser.
• The API: checks what you ask for and records it. The website and your own scripts use the same API.
• The control database: a PostgreSQL database of our own that holds every account and the desired state of every customer database.
• The orchestrator: reads the desired state and changes the servers until they match it.
• The servers that run your databases. The three a database relies on form its constellation.
• Around them: DNS for your hostname, TLS certificates, backups in object storage and monitoring.

The rule that holds it together: only the orchestrator changes servers, and it only ever does what the control database says should be true.

The website and the API

The website is a set of static files served by Cloudflare Pages. It holds no data. When it needs data it calls /api on its own address, and a small function on Pages forwards that request to our API. A problem with the website therefore never touches your databases: it holds nothing they need.

The API checks who you are, what your account may do, whether billing is in order and whether there is room for the database you asked for. If all of that passes, it writes the result to the control database and answers straight away. It never connects to a server that runs databases, and it never waits for one.

We built it this way so that an API request cannot fail halfway through a change to a server. Creating a database takes minutes; an HTTP request should take milliseconds. The API records the decision, and the slow part happens somewhere that can retry it. The website then polls the API and shows the setup phases as they complete, and anything the orchestrator is retrying.

The control database: desired state and the only queue

Each of your databases is a row in the control database. The row holds the desired state: the plan, the PostgreSQL version, the region, the replication mode and so on. It also holds a version number that goes up every time the desired state changes, and the version the orchestrator last made real. When the two numbers differ, there is work to do.

Longer jobs, such as a plan change, a restore or a password reset, are recorded as operations next to the database. An operation keeps its progress in the control database as it goes, so it can resume from where it stopped.

That is also the whole work queue. We run no message broker and no other service between the API and the orchestrator. A separate queue would be a second copy of the truth that could disagree with the first: a message lost, delivered twice or left behind after the row it described was deleted. With one database there is one place to back up, one place to look when something is stuck, and nothing to keep in step.

The orchestrator: reconciling until it matches

The orchestrator picks up a database that has work due, takes a short lease on it so no other copy of the orchestrator works on the same database, and runs its steps from the top. For a new database they are, in order:

1. Place: choose the servers, two in different sites for a high-availability plan.
2. Secrets: the owner password and the backup encryption key, both unique to this database.
3. Namespace: an isolated space in Kubernetes with its own resource limits and network rules.
4. Volumes: storage on the chosen servers, sized for the plan.
5. The PostgreSQL cluster: the primary and, on high-availability plans, the standby, run by Patroni.
6. Routing: every server learns your database's port and sends connections to the primary.
7. DNS: your hostname, checked until the name servers answer it.
8. First backup: a full backup, so the restore window starts before you write anything.
9. Billing: the database is added to your subscription.
10. Live: the status changes and the website shows the connection string.

Every step is idempotent: it first looks at what already exists and only does what is missing, so running it twice is the same as running it once. A step says it is done, or asks to be called again in a number of seconds, or reports an error. A temporary error is retried with a growing pause; a permanent one stops the database and pages us. The orchestrator commits after each step, so a restart in the middle of a create picks up at the first step that is not done yet.

Billing is a step for the same reason. It runs after everything else works, so a database that never goes live is never charged.

The same steps also repair things. A periodic check compares every live database with what is actually on the servers and in DNS. If something has drifted, it queues the database again and the steps put it back. Anything that exists on the servers but not in the control database is reported to us, never deleted automatically.

One request, end to end

Here is what happens when you create a database:

1. You choose a region and a plan and press create. The website sends the request to its own /api path.
2. The function on Cloudflare Pages forwards it to the API.
3. The API checks your account, billing and capacity, writes a new database row with its desired state, and answers with the database's id. If no servers have room, you get that answer now, before anything is created.
4. The website starts polling the database's status.
5. Within seconds the orchestrator claims the row and runs the steps above, committing after each one and waiting where a step has to wait, for example for PostgreSQL to start or for the first backup to finish.
6. When the last step is done, the database is live and the website shows the connection string and the password, once.

A plan change, a restore or a delete follows the same path: the API records the request, and the orchestrator carries it out step by step.

Servers, DNS, TLS, backups and monitoring

Databases run on servers we rent in separate sites, joined into one Kubernetes cluster (k3s) per region. The three servers a database relies on form its constellation; a later article covers how they vote, route and recover together. In short, each database is a PostgreSQL cluster managed by Patroni. On high-availability plans the primary and the standby are on two servers in different sites, and Patroni promotes the standby if the primary's server fails. Every server runs HAProxy, which listens on every database's port and passes each connection to whichever member is the primary at that moment.

• DNS: your hostname is served by Cloudflare and has one address for each server that holds a member of your database, with a 60-second TTL. Whichever address your driver picks, that server routes you to the primary.
• TLS: a certificate from Let's Encrypt covers every database hostname. HAProxy passes the encrypted connection through untouched, so it ends at PostgreSQL itself and your driver can verify the name with sslmode=verify-full. See Security.
• Backups: WAL-G copies every change and a daily full backup to object storage in the same region, encrypted with a key that belongs to that one database. We check every database's backups every few minutes and test restores automatically every week. See Backups and point-in-time restore.
• Monitoring: Prometheus collects metrics from every server and database, and alerts page us when a database is down, backups are late or a disk is nearly full. The same measurements feed the status page and the uptime shown on each database page.

What runs today

ElectraDB runs PostgreSQL: version 17 by default, and 18. Databases are in the EU region, with the US and Asia to follow. We are working on more database engines, and they will run on the same parts: the same API, control database, orchestrator and servers.