Choosing a Database Platform the Hard Way: What Months of Breaking Things on Purpose Taught Us

Where this started

It began with a simple-sounding question from our CIO's roadmap: can we run our databases on Kubernetes instead of virtual machines, and if we do, will they still be fast, scalable, and resilient enough to trust with production workloads?

That question sounds like it should have a quick answer. It doesn't. "Kubernetes" isn't one thing, it's a platform you build on top of, and the actual decision comes down to which database operator you run, how you configure it, and whether the promises those tools make actually hold up when something goes wrong. We decided the only honest way to answer that was to stop reading vendor marketing and start actually breaking things on a real cluster, on purpose, and writing down exactly what happened.

The starting point

We set up a dedicated Kubernetes cluster, Newton, managed entirely through GitOps, every change tracked, reviewed, and reproducible in Git rather than typed by hand into a terminal and forgotten. On top of it, we deployed two purpose-built database operators: CloudNativePG for PostgreSQL, and Percona XtraDB Cluster for MySQL, both genuinely production-grade, CNCF-adjacent projects, not experimental toys.

Then we ran them side by side against the same workloads on traditional virtual machines, to answer the first real question: does Kubernetes itself cost us anything in raw performance? The answer, reassuringly, was mostly no. PostgreSQL on Kubernetes landed within 15% of its VM equivalent once properly tuned. MySQL's story turned out to be more interesting, its clustering technology has a real, structural throughput ceiling under heavy concurrent load, and that ceiling exists on VMs too. It's the cost of keeping three copies of your data perfectly, safely synchronized, not a Kubernetes tax at all.

Moving past performance, into failure

Performance numbers are the easy part. The real test of a database platform is what happens when something breaks, and we made a deliberate decision early on: we wouldn't trust a platform's failure-handling claims until we'd broken it ourselves and watched it recover, or fail to.

Over the following weeks we ran a consistent, repeatable battery of tests against every platform we evaluated: kill the primary database server without warning and time how fast a replacement takes over. Kill enough servers at once to threaten data safety and see whether the system correctly refuses risky writes or dangerously accepts them. Simulate a genuine network split, not just a crashed server, and watch whether two servers can end up disagreeing about which one is really in charge. Deliberately corrupt a piece of the underlying storage while the database keeps running. Try to perform routine, planned maintenance and see whether the platform protects its most critical component or lets you accidentally take it down. And, critically, prove that backups actually restore correctly, not just that a backup command exits without an error.

That last one matters more than it sounds like it should. A backup you've never actually restored from is a belief, not a fact. We insisted on the actual, verified restore every time.

What we found, and why it mattered

The two purpose-built operators, CloudNativePG and Percona XtraDB Cluster, each handled failure the way you'd hope: automatic recovery, generally within tens of seconds, and correctly conservative behavior under the riskiest failure modes. But we also found something more interesting than either platform's individual strengths, we found that PostgreSQL and MySQL make genuinely different, deliberate safety tradeoffs by design. PostgreSQL-style replication stays available during a network problem but can silently lose the most recent writes if the primary dies at exactly the wrong moment. MySQL's stricter clustering refuses to accept any write it can't be certain is safe, at the cost of briefly refusing service. Neither choice is wrong. They're different answers to the same unavoidable tradeoff, and now we had real, measured evidence of both, not just documentation claims.

We also found our own mistakes, and they turned out to be some of the most valuable findings of the entire project. Early on, nearly every mysterious connectivity failure we hit for a full week traced back to one root cause: a network security rule we'd copied from an example, not understanding that Kubernetes silently locks a workload down completely the moment you apply even one narrow rule to it. That wasn't a flaw in the database tools. It was a real, transferable lesson about a genuinely non-obvious sharp edge in how the platform itself behaves, exactly the kind of thing that's expensive to learn in production and cheap to learn in a test.

Expanding the search

Two operators weren't enough to make a confident recommendation, so we kept going. We evaluated KubeBlocks, a newer platform that manages many different database engines through one consistent interface, and found a genuinely striking result: for routine, everyday configuration changes, the kind of thing that happens constantly in real operations, it was thirty to a hundred times faster than our original setups, with zero disruption. But it also had a real, disqualifying gap: its point-in-time recovery mechanism simply didn't work correctly on the version we tested, something we only discovered because we insisted on actually proving the restore, not just the backup.

We then evaluated StackGres, and it became, by the numbers, our strongest single recommendation: the fastest failure recovery of every platform tested, and the only one where backup and recovery worked correctly on the very first real attempt, no debugging required. It wasn't flawless either, it carries its own real, measured performance cost under heavy load, traced precisely to how it shares compute resources across several built-in helper processes bundled into every database instance, and it lacks the same automatic maintenance-safety protection our very first platform had by default.

Finally, we pushed into genuinely distributed databases, technology built to split data across many machines rather than just copy it for safety. Getting a real, working distributed PostgreSQL cluster running took real, hands-on troubleshooting most single-server setups never require, and running our standard benchmark against it produced a large, initially alarming performance gap. But digging into why revealed something important: our test was close to a worst-case scenario for that specific technology, not a representative one. The honest, useful finding wasn't "this is slower," it was "this pays a real coordination cost for a kind of workload it isn't meant to solve," a genuinely different, more useful piece of information for anyone actually deciding whether to use it.

Why this actually mattered

None of this was academic. Every one of these findings changes a real decision someone has to make: which platform to run, how to size it, what tradeoffs to accept knowingly instead of discovering by accident during an incident. The value of this work was never in any single benchmark number. It was in refusing to trust a claim, ours or any vendor's, until we'd watched it happen, and being honest in the write-up when a result surprised us, contradicted an earlier finding, or turned out to be our own mistake rather than the technology's.

That discipline, verify everything, write down the failures as carefully as the successes, and never let a clean story get in the way of what actually happened, is the real product of this project. The specific numbers will age. The habit of testing before trusting won't.