Page 2 of 3
A syscall (system call) is how a process asks the kernel to do something on its behalf: read a file, write to a socket, allocate memory, fork a child. Application code can't touch hardware or other processes directly, by design, so literally every meaningful interaction with the outside world eventually goes through one.
The default behavior of most I/O syscalls is blocking: when a process calls read() on a socket with no data available yet, that process simply stops running entirely and sits in a sleeping state until data shows up. This is efficient (it uses zero CPU while waiting) but means one thread can only ever be waiting on one thing at a time. Non-blocking I/O flips this: the syscall returns immediately, with either "here's your data" or "nothing's ready yet, ask again," which lets a single thread juggle thousands of open connections by checking on many of them in a loop instead of dedicating one thread per connection. This is the actual mechanical difference behind why an event-driven server like nginx or Node.js can handle enormous numbers of concurrent connections with a small number of threads, versus a traditional thread-per-connection server that needs one OS thread for every simultaneous connection, each with its own memory overhead and its own slot in the scheduler.
Since every meaningful interaction with the kernel goes through a syscall, tracing them directly is one of the most powerful ways to see what a process is actually doing, as opposed to what you assume it's doing.
strace -c -p 4821
# % time seconds usecs/call calls syscall
# ------ ----------- ----------- --------- ----------------
# 62.10 0.041233 8 5122 read
# 19.44 0.012901 41 312 futex
# 10.02 0.006650 110 60 connect
strace -c summarizes which syscalls a process is spending its time in and how often, rather than printing every single call live, which is usually the more useful first look. A process dominated by read calls against a file is doing exactly what it looks like: reading a lot of data. A process spending most of its time in connect might be struggling to reach a slow downstream service. A pile of time in futex (the syscall underlying most lock implementations) is a strong hint that threads are fighting over a lock rather than doing real work. The tradeoff to know going in: strace works by intercepting every syscall the target makes, which meaningfully slows the process down while you're attached, so it's a targeted diagnostic tool for a specific suspect process, not something you leave running on a healthy production service.
When a Linux system runs critically low on memory with nothing left to reclaim, the kernel doesn't just fail the next allocation and hope for the best; it invokes the OOM killer (out-of-memory killer), which picks a process to kill outright in order to free enough memory to keep the rest of the system alive. Which process gets picked is not random, and it isn't simply "whatever's using the most RAM," though that's the biggest factor.
cat /proc/4821/oom_score # the kernel's current computed "how killable is this" score
cat /proc/4821/oom_score_adj # a bias you (or root) can apply, from -1000 to 1000
echo -500 > /proc/4821/oom_score_adj # make this process much less likely to be picked
Every process gets an oom_score, computed mainly from how much memory it's using relative to the system, and the kernel kills whichever runnable process has the highest one. oom_score_adj lets you bias that calculation per process: a large negative value makes a process dramatically less likely to be chosen (-1000 makes it fully immune), while a positive value makes it more likely to be sacrificed first. This is exactly the mechanism container platforms use under the hood to protect critical system processes while letting a memory-hungry application container take the hit first. Finding Killed process 4821 (java) total-vm:... in dmesg or journalctl -k after an unexplained restart is the single most common way people first discover the OOM killer exists.
uptime and top both print a load average as three numbers, and it's one of the most consistently misread metrics in all of Linux operations.
opsquiz@devops-essentialsopsquiz@devops:~$ uptime14:32:01 up 3 days, 2:11, 1 user, load average: 8.42, 6.10, 4.55
Those three numbers are the average number of processes that were either running or waiting for a resource, over the last 1, 5, and 15 minutes respectively. The common misconception is that this is purely a CPU metric, comparable directly to CPU utilization percentage. It isn't: load average includes processes in uninterruptible sleep (the D state from the basic tier), meaning processes blocked waiting on disk I/O count toward load exactly the same as processes actually burning CPU cycles. A load average of 8 on an otherwise idle 4-core box that's stuck waiting on a slow disk is a completely different problem, with a completely different fix, than a load average of 8 because 8 CPU-bound processes are genuinely competing for 4 cores. The rising-1-minute, falling-15-minute pattern tells its own story too: a spike that's already recovering versus one still building.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.