Linux - Hard

Automating Configuration Idempotently

Hand-editing config files on production servers doesn't scale past a handful of machines, and it isn't repeatable - two servers configured "the same way" by hand over time reliably drift apart. Configuration management tools (Ansible, Chef, Puppet) fix this by describing the desired end state and letting the tool figure out what changes are needed to get there.

# Ansible task example
- name: ensure nginx is installed
  apt:
    name: nginx
    state: present
- name: ensure nginx config is correct
  copy:
    src: nginx.conf
    dest: /etc/nginx/nginx.conf
  notify: restart nginx

The key property is idempotency: running this playbook once, or a hundred times, against the same server produces the same result - it only changes what actually needs changing. This is what makes automated configuration trustworthy enough to run unattended and repeatedly, rather than being a script you're afraid to re-run because it might do something different the second time.

---

Incident Response Workflow on Linux Servers

Under pressure, having a repeatable order of operations matters more than knowing exotic commands. A reasonable default sequence when a Linux server starts misbehaving:

1. Confirm the symptom - is it actually down, or slow, or erroring for a subset of requests? curl it, check a health check endpoint, don't trust a single Slack report.
2. Check the obvious resources fast - uptime for load, df -h for disk, free -h for memory. This takes seconds and rules out entire categories of cause.
3. Check what changed recently - a deploy, a config push, a cron job that just ran. Most incidents trace back to a recent change, not spontaneous hardware failure.
4. Read the logs at the right time window - journalctl --since bounded tightly around when the symptom started, not an unbounded scroll through everything.
5. Form a hypothesis before acting - restarting a service can genuinely fix things, but it also destroys the evidence (stuck processes, memory state) you'd need to find the root cause. On a first occurrence, prefer capturing state (a strace, a heap dump, relevant log lines) before you restart, if the outage severity allows it.

That last point is a real tension every on-call engineer faces: the fastest fix is often "restart it," but the fastest fix and the fastest diagnosis pull in opposite directions. Deciding which one the situation calls for - and being honest about which one you're doing - is a big part of what separates reactive firefighting from a system that actually gets more reliable over time.