Monitoring & Observability Interview Questions & Answers

A liveness probe answers 'is this process stuck or deadlocked?', and failing it typically triggers a restart. A readiness probe answers 'can this instance currently serve traffic?', such as during startup while it's still warming a cache or waiting on a dependency, and failing it just removes the instance from load balancing without restarting it. Conflating the two is a common mistake that causes unnecessary restarts of instances that just need more time to warm up.

Downsampling reduces the granularity of data as it ages, aggregating raw high-resolution samples into coarser buckets (like hourly or daily rollups) for anything beyond a certain age, since long-term trend analysis rarely needs second-by-second precision. This keeps storage costs bounded while still preserving the ability to look at capacity trends or year-over-year comparisons, unlike simply deleting old data outright.

A trace's spans form a waterfall showing exactly how much of the total duration each hop and sub-operation consumed, and here the child span for one database query, 1750ms, is nearly the entire 2-second budget, making it the obvious bottleneck. The gateway, auth, and payment spans are all fast and irrelevant to this particular slowdown, so optimizing them wouldn't move the needle. This is exactly what makes tracing more useful than logs or metrics alone for this kind of question: it directly attributes where time went within one request, with no extra instrumentation needed beyond the spans already captured.

A runbook is a step-by-step reference tied to a specific alert or failure mode, written so that whoever is on call, even someone unfamiliar with that particular system, can follow a known-good procedure instead of improvising under pressure. Linking a runbook directly from the alert reduces both mean time to resolution and the cognitive load on the responder.

Prometheus's rate() function is reset-aware: when it sees the current value drop below the previous one, it assumes a counter reset occurred (rather than a real negative rate, which is impossible for a counter) and adjusts the calculation accordingly instead of producing a nonsensical negative spike. This is precisely why counters, paired with rate(), remain the standard way to track cumulative totals like request counts even across restarts.

Capacity planning uses historical trend data to forecast when a resource will run out under current growth rates, turning what would otherwise be a reactive, middle-of-the-night emergency into a scheduled, proactive piece of work with weeks of lead time. A single threshold alert at 95% only tells you the problem is imminent, whereas trend-based forecasting tells you it's coming weeks in advance, while there's still time to provision more capacity calmly.

Synthetic monitoring proactively simulates real user behavior from the outside on a schedule, so problems can be caught even when there's no real user traffic hitting the affected path at that moment. It's a form of black-box monitoring, since it checks observable outcomes without looking at the system's internals, which is exactly what makes it good at catching issues that internal metrics might miss.

Every unique combination of label values creates a separate time series in the underlying store, so a label like user_id with millions of possible values (high CARDINALITY) can multiply one metric into millions of series almost overnight, which is exactly what overwhelms storage and indexing. This is a cardinality problem, not a metric-type problem, switching to a gauge wouldn't fix it, and the limit that gets hit is on total time series or label combinations, not on the count of distinct metric names. The standard fix is to keep high-cardinality fields like user_id in logs or traces, where they belong, not as a metric label.

Dashboard variables let a single dashboard definition stay generic, with panel queries referencing a variable like $service instead of a hardcoded service name, and a dropdown at the top lets whoever's viewing it pick which specific value to plug in. This turns what would otherwise require 40 separate, hand-maintained dashboards into one reusable template, which is exactly the maintenance burden the scenario describes.

Traces stitch together spans from every service a single request touched, showing the request's path and per-hop timing, exactly what's needed to find where time was spent for one request. Metrics are great for aggregate trends and dashboards but discard per-request identity, and logs give rich detail per event without inherently linking events across services into one request's journey. Alerts aren't a pillar of observability at all, they're a downstream consumer of metrics, logs, or traces.