Operating Systems - Hard

cgroups: How Docker and Kubernetes Actually Enforce Limits

Every resource limit you set in a Dockerfile, a docker run flag, or a Kubernetes pod spec is implemented using a single underlying kernel mechanism: cgroups (control groups). There's no separate "container resource management" system; Docker and Kubernetes are, at the OS level, just processes that create and configure cgroups on your behalf.

docker run -d --memory=512m --cpus=0.5 nginx
# under the hood, this creates a cgroup and writes limits into it, roughly:
cat /sys/fs/cgroup/memory.max            # 536870912 (512m in bytes)
cat /sys/fs/cgroup/cpu.max                 # 50000 100000  (50ms of every 100ms window)

CPU limits work differently from memory limits in a way that trips a lot of people up. --cpus=0.5 doesn't mean "run at half speed"; it means the cgroup gets a quota: 50,000 microseconds of CPU time out of every 100,000-microsecond period, enforced across however many cores the container's threads happen to spread across. Burn through that quota early in the period and the kernel doesn't slow the container down proportionally, it stops scheduling it entirely until the next period starts, a phenomenon called CPU throttling. A container can be throttled hard while every dashboard shows CPU usage well under 100%, because usage is measured as an average and throttling happens in short, bursty windows that averages smooth right over. This is one of the most under-diagnosed sources of "why is this pod slow sometimes" in Kubernetes.

What Happens When a Cgroup Hits Its Memory Limit

Memory limits behave nothing like CPU limits. There's no equivalent of throttling for memory: a process either fits within its cgroup's memory ceiling or it doesn't.

opsquiz@devops-essentials
opsquiz@devops:~$ kubectl describe pod my-app-7d9f8
  Last State:     Terminated
    Reason:       OOMKilled
    Exit Code:    137

Exit code 137 isn't an arbitrary number; it's 128 + 9, signaling that the process was terminated by signal 9 (SIGKILL). When a cgroup's memory usage hits its configured limit, the kernel's cgroup-aware OOM killer looks inside that cgroup specifically and kills a process within it to bring usage back under the ceiling, using the same oom_score logic from the medium tier but scoped to just that cgroup rather than the whole machine. This is deliberately a separate, more localized event from the system-wide OOM killer: a single container hitting its own memory limit and getting killed shouldn't have to threaten every other container on the same node, and in a well-configured cluster it doesn't. The practical consequence is that "OOMKilled" in Kubernetes almost always means the container's own limit was too tight for what it actually needed, not that the node itself is out of memory, and the fix is usually to raise that specific limit or find and fix the leak, not to add more RAM to the host.

Namespaces: What Actually Isolates a Container

Cgroups limit how much of a resource a process can use. Namespaces control what a process can see in the first place, and they're the other half of what makes a container a container.

# roughly what a container runtime does when it starts a container
unshare --pid --mount --net --uts --fork /bin/sh
ps aux
# PID   USER  COMMAND
#   1   root  /bin/sh          <- this shell believes it's PID 1
hostname
# 4a9f2e1b3c8d                   <- its own hostname, unrelated to the host's

Each namespace type hides a different piece of the kernel's shared state. The PID namespace gives a container its own process ID numbering starting from 1, invisible to and from the host's real process tree. The network namespace gives it its own network interfaces, routes, and port space, which is exactly why ten containers on one host can each bind port 8080 without any conflict: from inside each one, it looks like the only process on the machine using that port. The mount namespace gives it its own view of the filesystem, so a container's / can be an entirely different filesystem tree than the host's /. The UTS namespace gives it its own hostname. None of this involves virtualizing the CPU or emulating hardware, which is precisely why containers are not VMs: a container is an ordinary Linux process, visible to and schedulable by the same single kernel as everything else on the host, just wrapped in namespaces that change what it can see and cgroups that limit what it can use. That shared-kernel reality is also why a kernel-level vulnerability is a much bigger deal for container isolation than it is for VM isolation, and why container escape exploits are specifically about breaking out of a namespace or cgroup boundary, not about breaking out of an emulated machine.