Monitoring & Observability Interview Questions & Answers

The four golden signals are latency, traffic, errors, and saturation, and saturation specifically measures how close a system is to its resource limits (CPU, memory, queue depth, and so on) before performance degrades. The other options describe engineering process or delivery metrics, not signals you'd track on a service's core health dashboard.

When one database goes down, every service that depends on it starts failing too, and each of those failures fires its own alert - so you get flooded with pages that are really all symptoms of the same root problem. Alert correlation groups related alerts together and only pages you for the root cause, so on-call gets one clear signal instead of a hundred confusing ones.

Dashboard variables let a single dashboard definition stay generic, with panel queries referencing a variable like $service instead of a hardcoded service name, and a dropdown at the top lets whoever's viewing it pick which specific value to plug in. This turns what would otherwise require 40 separate, hand-maintained dashboards into one reusable template, which is exactly the maintenance burden the scenario describes.

A counter is a cumulative metric whose value only ever goes up, or drops back to zero when the process restarts; it's the right choice for totals like requests served or errors raised. A gauge, by contrast, can go up or down (like current memory usage), which is why counters and gauges are treated differently by query functions like rate().

Log sampling deliberately keeps only a fraction of high-volume, low-value log lines (like routine successful requests) while still capturing everything from unusual or error paths, which are the lines most likely to matter during an investigation. This cuts storage and ingestion cost dramatically while preserving statistical visibility into normal behavior and full visibility into problems, which is a much better trade-off than either logging everything or turning logging off.

Toil is manual, repetitive, automatable operational work that doesn't require human judgment and scales linearly with load rather than getting easier over time, and the standard SRE response to identifying it is to automate it away rather than just reassign it to someone else. Since the recovery steps here are always identical and well-understood, this is close to a textbook case for turning the runbook into a self-healing automation instead of continuing to burn a human's time on it every week.

A correlation ID (sometimes called a request ID or trace ID) is generated once, typically at the entry point, and passed along with the request to every downstream service, which includes it in every log line it writes. An engineer can then search logs across every service for that one ID and reconstruct the full request path, without which same-timestamp logs from unrelated requests would be impossible to tell apart.

A normal alert only tells you something when a bad condition is detected, which is useless if the alerting system itself is the thing that's broken. A dead man's switch inverts this: it's an alert (or external check) configured to expect a regular 'I'm still alive' heartbeat, and it's the missing heartbeat itself that triggers the page, which is why it can catch failures in the monitoring pipeline that would otherwise monitor itself into silence.

Head-based sampling decides whether to keep a trace at the very start of a request, before its outcome is known, so it samples uniformly at random and is just as likely to throw away a slow, failing request as a fast, successful one. Tail-based sampling instead buffers spans until the request finishes, then applies rules (like 'always keep errors and high-latency requests') to bias sampling toward exactly the interesting cases, at the cost of needing to hold onto all spans until a request completes, which is more resource-intensive but far more useful for debugging.

INFO is meant for notable, expected events in normal operation, things a team might want visibility into without them signaling a problem. DEBUG is for fine-grained detail useful only during active troubleshooting, and WARN/ERROR are reserved for situations that are unexpected or require attention, which a successful checkout is not.