Longhorn: Persistent Storage for Databases on Kubernetes

Kubernetes was built with stateless applications in mind first. A Pod dies, a new one starts somewhere else, no problem, since nothing about the old Pod mattered once it was gone. Databases break that assumption completely: a Postgres or MySQL Pod that gets rescheduled to a different node still needs to come back up with the exact same data on disk. That's the problem persistent storage solves, and Longhorn is one of the most widely used answers to it.

Why local disk isn't enough

The simplest way to give a Pod storage is a hostPath volume, pointing directly at a directory on the node's local disk. It works, until the Pod gets rescheduled to a different node, at which point the data is still sitting on the old node and the new Pod starts with nothing. Local storage ties your data to a specific physical machine, which is exactly the kind of coupling Kubernetes is designed to avoid everywhere else.

What you actually want is storage that exists independently of any single node, so a Pod can be rescheduled anywhere in the cluster and still find its data waiting for it. That's what a distributed storage system provides, and Longhorn is a cloud-native one built specifically to run inside Kubernetes rather than as an external system you bolt on.

What Longhorn actually does

Longhorn is a distributed block storage system that runs as a set of Kubernetes-native components (deployed via a DaemonSet and a set of controller Pods) and exposes a standard CSI (Container Storage Interface) driver, so any workload just requests a PersistentVolumeClaim like it would with any other storage class.

Under the hood, when you create a volume, Longhorn creates multiple replicas of it (three, by default) spread across different nodes. Every write goes to all replicas. If the node holding the active replica goes down, Longhorn promotes one of the healthy replicas and the volume keeps working, with the workload none the wiser beyond a brief interruption while the failover happens.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 20Gi

That's the entire application-facing interface. A StatefulSet running Postgres references this PVC exactly like it would reference storage from any cloud provider's managed disk service, it has no idea Longhorn is doing replication underneath.

Snapshots and backups, not the same thing

Longhorn gives you two related but distinct safety nets:

• Snapshots are point-in-time copies stored on the same cluster, fast to create and fast to restore from, but they don't protect you if the whole cluster is lost.
• Backups ship a snapshot out to external object storage (S3, or an NFS share), so they survive even a total cluster failure. Backups take longer and cost more (network transfer, external storage), which is why they're usually scheduled less frequently than snapshots.

A reasonable pattern for a database is frequent snapshots (say, every few hours) for quick rollback during normal operations, plus a daily backup to S3 as the actual disaster recovery plan.

Where Longhorn helps, and where it doesn't replace database-level HA

This is the part that's easy to get wrong: Longhorn replicating your volume at the block level is not the same thing as database replication. Longhorn makes sure the bytes on disk survive a node failure. It has no idea whether those bytes represent a database in a consistent state at every instant, that's still entirely the database engine's job.

In practice, this means:

• Longhorn protects you from infrastructure failures (a node dying, a disk failing).
• Postgres streaming replication or MySQL group replication protects you from application-level issues and lets you actually scale reads across replicas, something block-level replication can't do for you.
• The two are complementary. A single-instance Postgres deployment on Longhorn-backed storage is far more resilient than one on plain local disk, but it's still a single point of query failure until you add real database replication on top.

The tradeoff nobody mentions upfront: latency

Distributed replicated storage has to write to multiple replicas over the network before it can acknowledge a write, compared to a local NVMe disk that just writes to itself. For most application workloads this overhead is unnoticeable. For write-heavy databases with tight latency requirements, it's worth benchmarking directly rather than assuming Longhorn behaves identically to local disk. Longhorn's own documentation is upfront about this: it's built for reliability and flexibility on commodity hardware, not to compete with a locally-attached NVMe drive on raw write latency.

A practical starting checklist

• Set replica count based on how many nodes you can tolerate losing at once, three is the sane default for most clusters.
• Use anti-affinity (Longhorn handles this automatically) so replicas of the same volume never end up on the same node, defeating the purpose of replication.
• Schedule both snapshots (frequent, cheap) and backups (less frequent, off-cluster) rather than relying on just one.
• Benchmark actual write latency for your specific database workload before assuming it's fine, don't guess.

Storage is one of those things that's invisible when it works and catastrophic when it doesn't. Getting comfortable with how Longhorn actually behaves, rather than treating it as a black box, is worth the time before you put a real database on top of it. If you want to test your grasp of the surrounding Kubernetes concepts, try the Kubernetes quiz on OpsQuiz.