One database, one tenant: how we keep databases apart

A tenant is one database

In most hosting, a tenant is a customer: everything one account owns sits together, and the walls are drawn around the account. We draw them around each database instead. Two databases of the same account are as separate from each other as two databases of different customers.

The reason is that a customer's databases are rarely one thing. One may be production and another a test copy, one may hold payments and another analytics. A runaway query, a wrong allowlist entry or a leaked password in one of them should not reach the others. Drawing every boundary around the same unit also keeps the system simple: creating, resizing, restoring and deleting a database each act on exactly one set of resources, and nothing is shared that could be left behind.

The rest of this article goes through those boundaries one layer at a time: compute, network, credentials, storage and backups, and finally what our own staff and software can see.

Compute: a namespace with fixed limits

Every database runs in its own Kubernetes namespace. Its members, its secrets and its certificate live there and nowhere else, and deleting the database deletes the namespace with everything in it.

Each namespace has a ResourceQuota and a LimitRange sized to the database's plan. Memory is requested and limited at the same value. That matters on a shared server: a process that may use more memory than it reserved takes it from whatever is free, and when the server runs short the kernel has to pick something to stop. With request equal to limit, the memory a database is promised is the memory it can use, and the sum of all promises on a server never exceeds what the server sells. One database growing cannot push another out of memory.

CPU works the same way on paid plans: the vCPUs in your plan are dedicated to your database, requested and limited at the same value. Dev and Dev+ share a pool of CPU between the databases on it, which is why they cost less and why they are meant for development and testing rather than production traffic.

Storage is local to each server and limited per database. Each member's volume is a directory on an XFS file system with its own project quota, so a database that fills its storage stops at its own limit and cannot fill the disk for its neighbours. See Plans, resizing and storage limits for what happens as a database approaches its size.

Network: closed by default

Every namespace starts with a default-deny network policy, so nothing can reach a database's members and they can reach nothing, until a rule allows it. On top of that, a Cilium network policy opens exactly what the database needs:

• In: only from HAProxy on the servers (your connections), from the database's own namespace (its two members replicating to each other), and from the operator that manages it.
• Out: only to the Kubernetes API (where Patroni keeps its leader lock), to DNS, to its own namespace, and to HTTPS for its backups.

A database cannot open a connection to another database's members, even on the same server, and it cannot reach arbitrary hosts on the internet.

Its own port and hostname

Each database has its own port, which HAProxy on every server listens on, and its own hostname, <id>.db.electradb.com. When a database is deleted, its port is held back for 30 days before it can be given to another database. Old clients, cron jobs and forgotten scripts keep trying a connection string for a while after its database is gone, and they should reach nothing rather than a new customer's database.

HAProxy decides where to send a connection by asking each member whether it is the primary. That check has to name the database, not just get a successful answer. Kubernetes reuses pod addresses: when a member stops, its address can be given to the next pod that starts, possibly another tenant's, which would answer the same question with its own yes. So HAProxy only counts a member as the primary when the answer comes from the right cluster; a yes from any other cluster counts as down.

Your own controls

You can limit who connects with a per-database IP allowlist of up to 50 addresses or ranges. Every connection uses verified TLS with a public certificate for your database's hostname, and our connection strings use sslmode=verify-full, so your client checks that it is talking to your database and not to anything else on the network. Both are described in Security.

Credentials: an owner, not a superuser

Each database has its own owner role, u_<id>, with its own generated password. It owns the app database and everything you create in it. It is deliberately not more than that:

• No superuser. A PostgreSQL superuser can read files on the server and run programs there, which would turn a database login into a shell inside the database's environment.
• No CREATEDB, so the role works in the database it was given.
• No untrusted languages, which can run arbitrary code with the server's permissions.
• No shell and no Kubernetes identity. There is no login to the servers, and nothing a customer holds lets them talk to the cluster.

The password is shown once, when the database is created or the password is reset. A leaked password opens one database, and resetting it closes that one database again without touching the others.

Backups and old copies

Each database's backups go to their own storage area in object storage, under their own prefix, and are encrypted with a key that belongs to that one database. The encryption happens on the server before upload, with libsodium, so the object storage only ever holds ciphertext. One database's key cannot decrypt another's backups. Details are in Backups and point-in-time restore.

A restore never writes into an existing database. It creates a new one, with its own namespace, port, hostname, password, backup area and encryption key. The restored data is the same; every boundary around it is new.

Local disks leave one more case: a volume whose database has moved away. When a server fails and we fence it, the database copies it held are rebuilt elsewhere. Before a fenced server may take databases again, every stale volume on it is deleted, so an old copy of your data cannot be served or read later by whatever is placed there next. The steps are in What happens when a server fails.

On our side

The same idea applies to our own software. The parts of ElectraDB that face the internet hold as little as they can:

• Database passwords, encryption keys and cluster credentials are encrypted at rest in our control database.
• The web service that answers the API and the website cannot read cluster credentials or backup keys at all. Only the orchestrator, which runs the databases, uses them.
• A database password can be revealed once, and that reveal is recorded in the audit log.
• Admin actions, whether through the API or our own tools, are recorded in the audit log.

The layers overlap on purpose. Code running inside one database has no network path to another; a client on the internet needs the password and, once you set an allowlist, an address on it; and even the owner of a database works without superuser powers. That set of boundaries, drawn around every single database, is what we mean by one database, one tenant. For how the pieces of the service fit together, see Inside ElectraDB.