Kubernetes - Hard

Observability

metrics-server is the minimal, cluster-wide component that kubectl top and HPA's CPU/memory scaling both depend on - it's not a full monitoring solution, just a lightweight aggregator of current resource usage, with no history retained.

Prometheus is the de facto standard for actual metrics collection and alerting at scale, typically paired with kube-state-metrics (which exposes the state of Kubernetes objects themselves as metrics - how many Pods are in CrashLoopBackOff right now, Deployment replica mismatches - distinct from cAdvisor-sourced container resource metrics, which measure actual CPU/memory/network usage).

Kubernetes audit logs record every request made to the API server - who did what, when, to which object - configured via an audit policy that controls verbosity per resource type (full request/response bodies vs just metadata). This is the forensic trail for "who deleted this Deployment at 3am," and it's off by default in many setups until explicitly configured.

---

Production Troubleshooting

The failure classes that specifically distinguish "knows Kubernetes" from "operates Kubernetes in production":

• Node NotReady - the kubelet has stopped reporting heartbeats to the API server: could be the kubelet process itself crashing, a network partition between the node and control plane, or the node genuinely out of resources. Pods already on that node aren't rescheduled immediately - there's a grace period (pod-eviction-timeout, default 5 minutes) before the control plane treats them as gone and reschedules replacements elsewhere.
• Node pressure (disk, memory, PID) - the kubelet actively evicts Pods to relieve it, following the same QoS-based ordering as memory eviction generally, and can mark a node unschedulable until pressure clears.
• kubelet/CNI failures - if the kubelet can't reach the container runtime or the CNI plugin fails to set up a Pod's network, that Pod is stuck ContainerCreating indefinitely - kubectl describe pod Events plus the kubelet's own logs on that node (not just kubectl output) are where the real answer lives.
• Cluster-wide outages almost always trace back to one of: etcd losing quorum, the API server itself being overwhelmed or down, or a botched control-plane upgrade - and the diagnostic order that actually works is control plane health first (can you even run kubectl get nodes?), then node health, then workload-level symptoms - never the reverse.

kubectl get nodes                              # control plane can still see nodes? Ready status?
kubectl get componentstatuses                  # legacy but still useful for a fast control-plane sanity check
kubectl -n kube-system get pods                 # control plane components' own health, if self-hosted
journalctl -u kubelet -f                        # (on a node) the kubelet's own logs, not just kubectl output