Page 2 of 3
Production hardening is about shrinking what can go wrong, not just reacting after it does.
Mandatory access control goes beyond regular Unix permissions: SELinux (RHEL family) and AppArmor (Debian/Ubuntu family) let you define exactly what a process is allowed to do - which files, which network actions - regardless of what user it's running as. A web server compromised through an application bug still can't read /etc/shadow or open arbitrary outbound connections if its SELinux/AppArmor profile doesn't permit it. This is precisely why disabling SELinux to "make an error go away" is a dangerous habit - it's usually blocking something for a real reason, and the fix is to adjust the policy, not remove the control.
getenforce # is SELinux enforcing, permissive, or disabled?
sestatus # detailed SELinux status
aa-status # AppArmor status equivalent
fail2ban watches logs (SSH auth logs especially) and automatically firewalls IPs that show a pattern of failed login attempts - a lightweight, automated defense against brute-force scanning that never sleeps.
Beyond tools, hardening is a mindset: minimize attack surface by not running services you don't need, keeping only necessary ports open, running services as unprivileged users, and keeping packages patched. Every open port, every running daemon, every piece of installed software is something that can eventually have a vulnerability - the smallest footprint that does the job is the safest one.
Once you have more than a couple of servers, tail -f on individual machines stops scaling - an incident spanning three servers means three terminals, three timezones of scrollback, and no way to correlate events across them.
Centralized logging ships logs off each host to one aggregated place - common stacks include the ELK stack (Elasticsearch, Logstash, Kibana) or lighter alternatives like Loki. On the host side, this usually means a lightweight shipper:
# example Filebeat-style config concept
paths:
- /var/log/nginx/*.log
- /var/log/myapp/*.log
output: elasticsearch (or a log aggregator endpoint)
The operational payoff is real: you can search "show me every ERROR across every server in the last 15 minutes" as a single query instead of fifteen separate SSH sessions, and you retain logs even after the server that generated them is gone - which matters enormously with autoscaling or immutable infrastructure, where an individual instance might not exist anymore by the time you go looking for its logs.
A single server is a single point of failure, no matter how well-tuned. High availability is about designing so that no single failure takes the whole system down.
Health checks: a supervisor (a load balancer, an orchestrator, or systemd itself) periodically checks whether a service is actually working - not just "is the process running" but "is it responding correctly." A process can be alive and completely unable to serve a single valid request; only an active health check catches that.
Watchdogs: systemd supports a watchdog mechanism where a service must periodically "check in" (sd_notify(WATCHDOG=1)) or systemd assumes it's hung and restarts it - catching the specific failure mode where a process is technically still running but has deadlocked and stopped making progress, which a simple "is the PID alive" check would miss entirely.
[Service]
WatchdogSec=30
The core lesson at this tier: "is it running" and "is it working" are different questions, and conflating them is a common cause of outages lasting far longer than they should - dashboards showing green while users see errors.
Beyond a single disk, production storage decisions have real consequences for both performance and durability.
LVM (Logical Volume Manager) adds a flexible layer between physical disks and filesystems - you can resize volumes, add disks, and take snapshots without unmounting anything, which matters enormously when you can't just take a production database offline to grow its disk.
RAID combines multiple physical disks for redundancy, performance, or both - RAID 1 mirrors data across disks (survives one disk failure, no speed gain), RAID 5/6 stripe with parity (better space efficiency, survives one or two disk failures, slower writes due to parity calculation), RAID 10 stripes mirrored pairs (fast and resilient, but you sacrifice half your raw capacity). None of these are backups - RAID protects against a disk failing, not against someone running rm -rf on live data or a bug corrupting rows across every mirror simultaneously.
Filesystem choice matters too: ext4 is a solid, boring default; xfs handles very large files and high-concurrency workloads better, which is why it's common on database servers; zfs and btrfs add built-in snapshots and checksumming at the cost of more operational complexity. The right choice depends on the actual workload, not a general "best" filesystem.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.