Kubernetes

Kubernetes - Advanced

Page 2 of 4

High Availability

Control plane HA means multiple API server instances (behind a load balancer) and an etcd cluster with real quorum tolerance (3 or 5 members). Losing a single control plane node in an HA setup shouldn't cause any user-visible impact - the load balancer routes around it and etcd retains quorum - which is precisely the property a single-control-plane-node cluster doesn't have.

Multi-AZ clusters and zone-aware scheduling (via topology spread constraints, and topology-aware storage provisioning) are what actually make "HA" mean something at the workload level, not just the control-plane level - a Deployment with no anti-affinity or spread constraints can have every single replica land in one AZ, and that AZ going down takes the whole app with it despite "having 5 replicas."

Pod Disruption Budgets (PDBs) protect availability during voluntary disruptions specifically - node drains for maintenance, cluster-autoscaler scale-downs, manual kubectl drain - by declaring a minimum number/percentage of replicas that must stay available; the eviction API refuses an eviction that would violate the budget. Crucially, a PDB does nothing for involuntary disruption (a node crashing outright) - it only constrains disruptions initiated through the eviction API in the first place.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web

Multi-Cluster Kubernetes

Running multiple clusters (per region, per environment, or for blast-radius isolation) introduces problems a single cluster never has: multi-cluster service discovery (how does a Pod in cluster A reach a Service in cluster B - typically requires a purpose-built mesh or gateway layer, since Kubernetes' own Service discovery is cluster-scoped by design), and workload migration/failover (moving traffic and eventually workloads from one cluster to another during an incident or planned maintenance).

A hub-and-spoke fleet-management model - one central "hub" cluster (or control-plane-as-a-service) managing configuration and policy pushed out to many "spoke" workload clusters - is the common pattern once you're managing more than a handful of clusters by hand stops being tractable; it's the same underlying idea as centralized GitOps, just applied at the fleet level instead of the single-cluster level.


GitOps

GitOps applies Kubernetes' own reconciliation philosophy to deployment itself: a Git repository is the declared desired state for what should be running, and a controller (Argo CD, Flux) continuously reconciles the live cluster to match it - the same watch-diff-act loop every built-in controller runs, just pointed at a Git repo instead of etcd's own object store.

Drift detection is what falls naturally out of this model: if someone runs kubectl edit directly against a GitOps-managed resource, the controller notices the live state no longer matches Git and either flags it or - depending on configuration - actively reverts it back to match Git. This is a feature, not a bug, but it's a common point of confusion for anyone used to imperative kubectl habits on a GitOps-managed cluster, and "the GitOps controller keeps reverting my manual change" is really just the system doing exactly what it's designed to do.

Progressive delivery (canary releases, automated rollback on failing metrics) is commonly layered on top of GitOps tooling (Argo Rollouts, Flagger) - shifting a small percentage of traffic to a new version, watching real metrics, and automatically completing or aborting the rollout based on what's observed, rather than a rollout being purely time-based or all-or-nothing.

Secrets in GitOps are a genuine unsolved-by-default problem: you can't commit plaintext secrets to Git, so GitOps setups either commit encrypted secrets (Sealed Secrets, SOPS) that only the cluster can decrypt, or reference an external secret store and let a controller (External Secrets Operator) sync real values into the cluster at apply-time, keeping the actual secret material out of Git entirely either way.


Operators & CRDs

An Operator is a CRD plus a custom controller that encodes real operational knowledge about a specific piece of software - not just "create this Deployment" but "when the primary database Pod fails, promote a replica, update the Service to point at the new primary, and do it in the right order." The value of the Operator pattern is exactly this: turning manual runbook steps into automated, continuously-enforced reconciliation.

Finalizers let a controller intercept deletion: an object with a finalizer isn't actually removed from etcd until the finalizer is cleared, giving a controller a guaranteed chance to run cleanup logic first (deprovision a cloud resource the CR represents, say) before the Kubernetes object itself disappears. OwnerReferences establish parent-child relationships between objects (a ReplicaSet owned by a Deployment, a Pod owned by a ReplicaSet) - deleting the owner triggers garbage collection of everything it owns, which is the actual mechanism behind "deleting a Deployment cleans up its Pods."

A misbehaving Operator - stuck reconciling, deadlocked on a finalizer that never clears - is a distinct and recognizable failure class from ordinary workload issues: kubectl get <custom-resource> -o yaml and checking its status conditions plus the operator controller Pod's own logs is the diagnostic path, not treating it as a generic Pod problem.


    Welcome to OpsQuiz!

    Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.