Page 2 of 3
The basic tier introduced file descriptors; in production, the number of them a process is allowed to hold open at once is a hard, configured limit, and hitting it is one of the most common self-inflicted outages there is.
ulimit -n # soft limit on open file descriptors for the current shell
# 1024
cat /proc/4821/limits | grep "open files"
# Max open files 1024 4096 files
# raising it for a systemd-managed service, not just the current shell,
# by adding this to the unit file:
# [Service]
# LimitNOFILE=65536
A default soft limit of 1024 sounds generous until you remember that every inbound connection, every outbound connection to a database or cache, every open log file, and every pipe all consume one, and a busy service handling thousands of concurrent connections can burn through 1024 in seconds under real load. The failure mode is brutally simple and very recognizable in logs: EMFILE, "too many open files," at which point the process can't accept new connections, can't open new files, and often can't even open a log file to record what's happening to it, which is what makes this particular incident so disorienting the first time you hit it. ulimit set in a shell only affects processes launched from that shell; a service started by systemd or a container runtime needs its limit raised in its own unit file or container config, which is the detail that catches people who "already fixed this" in their SSH session and can't understand why the service is still hitting the ceiling.
When multiple processes are issuing disk reads and writes at once, the kernel's I/O scheduler decides the order those requests actually reach the device, and which scheduler is active has real consequences for latency-sensitive workloads like databases.
cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber
On spinning disks, where physically moving the read head is the expensive part, schedulers historically optimized heavily for reordering requests to minimize seek distance, at the cost of making some requests wait longer than others. On modern NVMe SSDs, seek time is essentially not a factor, and none (no reordering at all, requests pass straight through) is frequently the right choice, because the scheduler's own bookkeeping overhead can cost more than it saves. mq-deadline guarantees every request gets serviced within a bounded time, which matters for a database that needs consistent write latency for its transaction log far more than it needs raw throughput. Getting this wrong doesn't cause an outage; it causes a database with a stubbornly high p99 write latency that no amount of query tuning fixes, because the bottleneck was never the query, it was requests sitting in a scheduler queue behind other I/O that had been reordered ahead of them.
A short list of sysctl parameters covers the overwhelming majority of production kernel tuning a DevOps engineer actually does.
sysctl net.core.somaxconn # max queued incoming connections waiting to be accepted
sysctl vm.swappiness # 0-100, how eagerly the kernel reaches for swap
sysctl fs.file-max # system-wide ceiling on total open file handles
# persisted across reboots in /etc/sysctl.d/99-tuning.conf:
net.core.somaxconn = 4096
vm.swappiness = 10
fs.file-max = 200000
net.core.somaxconn caps how many fully-established connections can sit in the accept queue waiting for the application to call accept() on them; set it too low on a high-traffic service and connections get silently dropped during traffic spikes, before you're anywhere near out of CPU or memory, which makes the dashboards look fine while real clients are failing to connect. vm.swappiness, already touched on in the medium tier, is the difference between a kernel that proactively swaps out idle memory at the default of 60 (fine for a desktop, generally a mistake on a latency-sensitive service) versus one set low, like 10 or even 1, that fights much harder to keep everything in RAM before ever touching swap. fs.file-max is the whole-system version of the per-process ulimit -n ceiling from earlier; you can raise every individual service's own file descriptor limit and still hit a wall if the system-wide total is set too low for a machine running many such services at once.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.