Kubernetes - Hard

CNI & Networking Internals

The CNI (Container Network Interface) is the plugin contract that actually wires up a Pod's network namespace - Kubernetes itself doesn't implement Pod networking, it delegates to whichever CNI plugin is installed (Calico, Cilium, Flannel, cloud-native ones). At the Linux level, each Pod gets its own network namespace, connected to the node via a veth pair (one end in the Pod's namespace, one on the node), typically attached to a Linux bridge or routed directly, depending on the CNI's model.

Overlay vs routed networking is the fundamental architectural split between CNI plugins: an overlay network (e.g. VXLAN) encapsulates Pod traffic inside node-to-node tunnels, working on any underlying network topology at the cost of encapsulation overhead; a routed/BGP-based CNI instead programs real routes so Pod IPs are directly routable across the underlying network, avoiding encapsulation overhead but requiring more control over the underlying network.

kube-proxy is what actually implements Service load-balancing on each node, historically via iptables rules (a chain of NAT rules matching a Service's virtual IP and rewriting it to a real Pod IP - simple but scales poorly with very large numbers of Services) or the newer IPVS mode (a real in-kernel load balancer, built for exactly this problem, with far better performance at scale). Increasingly, eBPF-based dataplanes (Cilium being the flagship example) replace kube-proxy entirely, implementing Service routing directly in the kernel with lower overhead than either iptables or IPVS.

kubectl get pods -n kube-system -l k8s-app=kube-proxy   # confirm which component is actually running
iptables -t nat -L | grep KUBE                            # (on a node) see the actual NAT rules kube-proxy programmed

---

Ingress & Gateway API

An Ingress Controller's job, concretely: watch Ingress objects via the API server, and translate them into real reverse-proxy configuration (nginx.conf, Envoy config, whatever the controller is built on). TLS termination at the Ingress means the controller holds the certificate and decrypts HTTPS traffic before it reaches your Pods (which then usually just speak plain HTTP internally) - the standard pattern, since managing TLS certs at every individual Pod is unnecessary overhead.

Gateway API is the newer, more expressive successor Ingress is gradually being superseded by - it splits configuration into separate roles that map better to real organizational boundaries: a GatewayClass (infrastructure-provided, defines what kind of load balancer/proxy this is), a Gateway (a specific listener - which ports, which TLS certs), and HTTPRoute objects (application-team-owned routing rules attached to a Gateway) - letting platform teams and application teams manage their own layer without stepping on each other's config, which plain Ingress's single flat object never cleanly supported.

---

NetworkPolicy - Advanced

The single highest-leverage NetworkPolicy pattern in production: a default-deny policy per namespace, then explicit allow rules layered on top.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
  # no rules specified = deny everything not explicitly allowed elsewhere

Rules can match by pod selector (specific Pods), namespace selector (anything in a matching namespace), or IP blocks (ipBlock: {cidr: ...} - for traffic to/from outside the cluster entirely). Egress rules matter just as much as ingress: without an explicit egress allow for DNS (UDP/TCP port 53 to kube-dns), a default-deny-egress policy will silently break DNS resolution for every Pod it applies to - one of the most common NetworkPolicy debugging traps.

Debugging a NetworkPolicy issue always starts the same way: confirm the CNI plugin actually enforces NetworkPolicy at all (not every one does), then check for an overly broad default-deny with a missing allow rule, before assuming anything more exotic is wrong.

---

Storage - Advanced

The CSI (Container Storage Interface) architecture splits into components with distinct jobs: a CSI controller plugin handles cluster-level operations (provisioning/deleting volumes, attaching/detaching them from nodes), while a CSI node plugin (running on every node, usually as a DaemonSet) handles the actual mounting into a Pod's filesystem.

Volume lifecycle, precisely: provision → attach (to a node) → mount (into a specific Pod's filesystem) → unmount → detach → delete. A stuck volume almost always means one specific stage failed - kubectl describe pod/pvc and the CSI driver's own controller/node Pod logs will show which stage.

Multi-zone storage is a real constraint most people hit eventually: a zonal block storage volume can only attach to nodes in its own zone - a Pod using such a PVC that gets rescheduled to a different zone will simply fail to mount, since the volume can't follow it there. This is exactly why StatefulSets combined with zonal storage need topology-aware scheduling (volumeBindingMode: WaitForFirstConsumer on the StorageClass) so the PV is only provisioned in the same zone the Pod actually gets scheduled to, not chosen first and then hoped to match.

---