Page 2 of 2
Configuration management (hard tier) reduces config drift but doesn't eliminate it entirely - you're still mutating a long-lived server over time, and small differences accumulate: a manual fix here, a package upgraded slightly out of band there.
Immutable infrastructure takes a harder line: servers are never modified after deployment at all. Instead, you bake a complete, versioned golden image (with tools like Packer) containing the OS, all dependencies, and your application, and any change - a config tweak, a security patch, a new app version - means building a new image and replacing the running instances wholesale, never patching in place.
# conceptual Packer build step
packer build -var 'app_version=2.3.1' golden-image.pkr.hcl
The tradeoff is genuinely two-sided, not a free upgrade. You gain: every instance is provably identical to every other (it's literally the same image), rollback is just "redeploy the previous image" rather than trying to undo a sequence of live changes, and "works on this server" ambiguity disappears because there's no server-specific state to diverge. You lose: image builds and full-fleet rollouts take longer than a targeted config push, and it demands genuinely stateless application design - anything a service needs to persist has to live outside the instance (a database, object storage), because the instance itself is expected to be destroyed and replaced without ceremony.
At small scale, "add more CPU when it's slow" is a fine strategy. At fleet scale, capacity planning becomes a real discipline with real cost consequences, because over-provisioning by even a small percentage across thousands of instances is a large, ongoing dollar figure.
This means tracking actual utilization trends, not peak headroom on a handful of machines - p50/p95/p99 CPU and memory across the fleet over weeks, not a snapshot from top on one box. It means understanding the difference between vertical scaling (bigger machines - simpler, but has a hard ceiling and doesn't improve fault isolation) and horizontal scaling (more machines - has near-limitless ceiling and improves fault tolerance, but demands your application actually be able to run as multiple independent instances behind a load balancer).
cgroups resource limits (this tier, earlier) become a governance tool at this scale, not just a per-service safety net: setting sane defaults fleet-wide prevents one noisy or leaking service from starving its neighbors on a shared host, which is the difference between one team's bug causing one team's outage versus one team's bug taking down everyone sharing that hardware.
Beyond a single datacenter, you're managing failure domains at a geographic level. The core design questions change shape: not "will this server fail" but "will this entire region become unreachable" - a real, recurring event at cloud-provider scale (a regional network partition, a power event, a botched provider-side maintenance).
Design patterns at this level include active-active (multiple regions simultaneously serving live traffic, with data replicated and reconciled between them) versus active-passive (one region live, another kept warm and ready to take over). Each has real costs: active-active demands your application and data layer handle multi-writer conflict resolution correctly, which is a hard distributed-systems problem in its own right; active-passive is simpler to reason about but means paying for standby capacity that does nothing most of the time, and a failover path that's rarely exercised is a failover path you can't fully trust until you've actually tested it under real conditions.
Configuration management and golden images (both this tier) become the mechanism that keeps regions consistent with each other - without them, "region B" quietly becomes a slightly different, untested system from "region A," and you only discover the difference during the failover you needed to actually work.
At organizational scale, "secure" has to be demonstrable to auditors, not just true in practice - SOC 2, PCI-DSS, HIPAA, and similar frameworks require you to prove specific controls are actually enforced, continuously, across the fleet.
This turns several earlier-tier practices into compliance mechanisms. Immutable infrastructure (this tier) makes patch compliance simple to prove: every running instance was built from a specific, dated image, so "when was this patched" has an exact, auditable answer instead of "sometime, on some servers, probably." Centralized logging (hard tier) becomes the audit trail - who accessed what, when - and typically needs to be tamper-evident and retained for a defined period, not just useful for debugging.
Audit tooling like auditd records security-relevant kernel events - file access, privilege escalation, command execution by specific users - in a way that's itself protected from casual tampering:
auditctl -w /etc/passwd -p wa -k passwd_changes # watch for writes/attribute changes to /etc/passwd
ausearch -k passwd_changes # search recorded events by that tag
The judgment call at this tier is proportionality: logging and auditing everything, everywhere, at maximum verbosity generates enormous cost and noise without necessarily improving your actual security posture. Good compliance design means choosing which controls genuinely reduce risk or satisfy a real requirement, instrumenting those well, and resisting the temptation to check every possible box uniformly regardless of whether it earns its cost - the same fleet-wide cost discipline that shows up in capacity planning applies here too, just measured in audit and storage overhead instead of compute.
Real scenario-based DevOps questions, hands-on practice, and clear explanations for every answer.