Page 1 of 3
Running services reliably in production means writing your own unit files, not just calling systemctl on things someone else packaged.
# /etc/systemd/system/myapp.service
[Unit]
Description=My App
After=network.target postgresql.service # start only after these are up
Requires=postgresql.service # hard dependency - fail if this isn't available
[Service]
ExecStart=/usr/bin/myapp --config /etc/myapp/config.yml
Restart=on-failure # restart automatically if it exits non-zero
RestartSec=5
User=myapp # never run production services as root unnecessarily
MemoryMax=512M # hard resource cap enforced via cgroups
CPUQuota=50%
[Install]
WantedBy=multi-user.target
systemctl daemon-reload # required after editing any unit file, or systemd won't see the change
systemctl restart myapp
After= and Requires= are easy to conflate but answer different questions: After= is purely about ordering (start this after that one, but don't actually require it to succeed), while Requires= is about dependency (if that one fails or stops, this one gets pulled down too). Getting this wrong is a classic cause of boot-order flakiness - a service that "sometimes" comes up broken depending on how fast other services happened to start that particular boot.
Restart=on-failure matters enormously at scale: without it, a service that crashes just stays dead until a human notices and intervenes. With it, transient failures self-heal. But it's not free - a service that crash-loops rapidly can mask a real problem behind an endless restart cycle, which is why pairing it with RestartSec and alerting on restart counts, not just downtime, matters.
When a server is "slow," you need tools that tell you which resource is actually constrained, not guesswork.
uptime # load average: 1, 5, 15 minute averages of runnable+waiting processes
vmstat 1 # per-second snapshot of CPU, memory, and I/O activity
iostat -x 1 # per-second disk I/O stats, per device, with %util
strace -p 1234 # trace every system call a running process makes - slow but revealing
lsof -p 1234 # list every file/socket a process currently has open
Load average is one of the most commonly misread numbers on Linux. It is not CPU utilization - it's the average number of processes that are either running or waiting for a resource (CPU, but also disk I/O). A load average of 8 on a 4-core machine genuinely means something different depending on whether those 8 processes are waiting on CPU (you're CPU-bound, add cores or optimize) or waiting on disk I/O (you're I/O-bound, and more CPU cores won't help at all). vmstat's wa column (time spent waiting on I/O) is what disambiguates this - high load with high wa points at storage, not compute.
strace is a last resort, not a first one - it works by intercepting every system call, which can slow the target process down significantly. It's the right tool when you genuinely don't know where time is going (is it stuck in a read()? a network call? a lock?) but reach for lighter tools like top and iostat first.
The kernel exposes hundreds of tunable parameters at runtime through /proc/sys, managed with sysctl.
sysctl vm.swappiness # read a current value
sysctl -w vm.swappiness=10 # change it now, temporarily
# /etc/sysctl.d/99-custom.conf - persists across reboots
net.core.somaxconn = 4096 # max queued connections waiting to be accept()ed
net.ipv4.tcp_fin_timeout = 30 # how long a closed connection stays in TIME_WAIT
vm.swappiness = 10 # how aggressively the kernel swaps memory to disk
fs.file-max = 100000 # system-wide max open file handles
Each of these is a real production tradeoff, not just a knob to max out. vm.swappiness at the default (60) means the kernel will proactively swap out idle memory even when RAM isn't under pressure, trading a small amount of latency-on-access for keeping more page cache available - fine for a desktop, often actively harmful for a latency-sensitive database server, where you'd rather lower it aggressively and accept less cache. net.core.somaxconn too low on a high-traffic web server causes connections to be silently dropped during traffic spikes, well before you're actually out of CPU or memory - a server can look perfectly healthy on every dashboard while quietly rejecting connections because a queue depth was set for a much smaller workload.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.