Kubernetes - Advanced

Large-Scale Kubernetes

Clusters in the 1000+ node range hit specific, well-known scaling walls: API server load grows with both node count and total object count (a cluster with far more objects per node than typical stresses the API server disproportionately to raw node count alone), etcd needs proportionally more disk/network headroom as total object count grows, and the scheduler and controllers all do proportionally more work per reconciliation cycle as there's simply more to reconcile.

IP address exhaustion is a very real, concrete constraint at scale: each node is typically allocated a fixed-size CIDR block for its Pods, and running out of allocatable Pod IPs (either per-node or cluster-wide) hard-blocks new Pod scheduling on an otherwise-healthy node - a capacity planning problem that has nothing to do with CPU/memory and everything to do with the cluster's networking configuration.

Cluster sharding (splitting what would be one enormous cluster into several smaller ones, workload-partitioned) is a legitimate large-scale strategy specifically because a single cluster's control plane has real ceilings - past a certain size, "one bigger cluster" stops being the right axis to scale on at all, and "more clusters" becomes the more tractable answer.

---

Advanced Reliability / SRE

SLOs and error budgets, applied to a Kubernetes platform specifically: define an availability target for the platform and for individual critical workloads, then track the error budget being consumed by real incidents - and let that budget consumption, not gut feeling, drive the tradeoff between shipping changes faster and slowing down to stabilize.

Failure-domain design means deliberately reasoning about what actually shares a failure boundary - a single AZ, a single node pool, a single etcd cluster, a single ingress controller instance - and ensuring genuinely critical workloads don't have every replica sharing the same one, the same discipline as multi-AZ scheduling but applied at the whole-platform level, not just one Deployment.

Chaos engineering (deliberately injecting failure - killing a random Pod, partitioning a network link, killing a node) validates that your actual failure-recovery mechanisms (PDBs, anti-affinity, HPA, liveness probes) work the way you believe they do, under real conditions, rather than trusting the YAML alone - a game day is the practiced, planned version of this, done deliberately rather than waiting for production to teach you the same lesson unplanned.

RTO/RPO (Recovery Time Objective / Recovery Point Objective) are the concrete numbers a disaster-recovery plan is actually built around: how long can the platform be down, and how much data can be lost, in the worst case - etcd backup frequency, cross-region cluster standby strategy, and incident-response runbooks should all trace back to explicit RTO/RPO targets, not be designed in the abstract.

---

Real-World Architecture Scenarios

The judgment-call category - these are the questions that separate "knows the terminology" from "would actually know what to do":

• All Pods stuck Pending cluster-wide - check cluster-level capacity first (are nodes actually full, or is the Cluster Autoscaler failing to add more?), then whether it's scoped to one workload (a bad nodeSelector/affinity) versus genuinely every Pod (a scheduler or API server problem) - the scope of the symptom tells you which layer to investigate first.
• API Server latency suddenly increases - correlate against etcd health (leader elections? disk I/O latency?) before assuming the API server itself is the bottleneck; a very common root cause is a client (a misbehaving controller, a CI pipeline) hammering it with an unbounded request rate.
• etcd disk reaches 100% - immediate risk of etcd refusing all writes; the fix is compaction/defragmentation plus addressing whatever's driving unusually high object churn, not just adding disk and moving on.
• Pods communicate within a node but not across nodes - almost always a CNI-level problem (overlay/routing misconfiguration between nodes specifically), not an application or Service issue - test with raw Pod-to-Pod connectivity across nodes before looking at Services at all.
• A Service works internally but not externally - isolate layer by layer: are Endpoints populated correctly (internal problem, not external)? Is the Ingress/LoadBalancer's own health check passing? Is DNS resolving to the right external IP? Each is a genuinely different failure with a different fix.
• HPA isn't scaling - check whether metrics-server (or the custom metrics adapter) is actually reporting data at all first; a shockingly common cause of "HPA does nothing" is simply that the metrics pipeline it depends on is broken or missing, not that the HPA's thresholds are wrong.
• A GitOps controller keeps reverting a change - this is the system working as designed against a manual out-of-band edit, not a bug - the actual fix is updating Git, not fighting the controller.
• A Kubernetes upgrade breaks workloads - almost always a removed/changed API version some manifest or Helm chart still targets - check deprecation notices for the exact target version before upgrading, not after something breaks.
• One AZ goes down - this is exactly the scenario multi-AZ topology spread constraints and PDBs are for; if it takes the whole application down anyway, that's a signal replicas were never actually spread the way you assumed.