Question 1

You want to track the current number of items sitting in a queue, a value that goes up when items arrive and down as they're processed. Which metric type should you use?

Accepted Answer

Gauge, since it represents a value that can arbitrarily go up or down. A GAUGE is the right choice for any value that can move in both directions, like queue depth, memory usage, or active connections. A counter can only increase, or reset to zero on restart, so it can't represent a value that shrinks, like a queue draining. Histograms and summaries are for observing the distribution of a stream of values, such as request durations, not for tracking one current point-in-time value.

Question 2

An engineer is graphing the request rate of an HTTP counter metric using PromQL. Using irate() over a 5 minute range produces a jagged, spiky line, while rate() over the same range produces a smooth trend line. Why?

Accepted Answer

irate() calculates the rate using only the last two data points in the range, making it react instantly to short spikes, while rate() averages the increase over the entire range, smoothing out noise. irate() (instant rate) computes the per-second rate using just the two most recent samples in the range, which makes it very responsive to sudden changes but noisy on a graph. rate() averages the increase across the whole range vector, producing a smoother trend that's better suited for alerting and dashboards, while irate() is more useful for zooming into fast-moving, recent behavior on a fine-grained graph.

Question 3

During an incident postmortem, a manager asks the writer to name the engineer who pushed the faulty config, so it's clear whose mistake caused the outage. What does a blameless postmortem culture say about this request, and why?

Accepted Answer

The postmortem should focus on the contributing systemic factors, like why the faulty config could reach production unreviewed, rather than which individual pushed it. A blameless postmortem is written on the assumption that people acted reasonably given the information and tools they had at the time, so the value comes from asking systemic questions, like why a bad config could be deployed without review or a safety check catching it, since fixing those prevents the NEXT incident regardless of who was on call. Naming and focusing on an individual instead tends to make engineers hide mistakes or route around monitoring in future incidents, which produces worse data and worse outcomes over time. It doesn't mean skipping the postmortem or refusing to describe what happened, it means describing what happened without assigning personal fault.

Question 4

A page fires for 'disk usage above 90% on db-primary.' Attached to the alert is a link to a document with step-by-step instructions for how to safely free up space on that specific server. What is this document called?

Accepted Answer

A runbook. A runbook is a step-by-step reference tied to a specific alert or failure mode, written so that whoever is on call, even someone unfamiliar with that particular system, can follow a known-good procedure instead of improvising under pressure. Linking a runbook directly from the alert reduces both mean time to resolution and the cognitive load on the responder.

Question 5

A latency dashboard shows a sudden, sustained jump in error rate at 2:14 PM. Separately, the team's deploy pipeline shows a release went out at 2:12 PM. Nobody made this connection until a customer complaint an hour later, because the two systems aren't linked. What practice would have surfaced this connection immediately on the dashboard itself?

Accepted Answer

Adding deployment events as annotations (vertical markers) directly on the relevant dashboards, so a spike lining up with a recent release is visually obvious. Many dashboarding tools (like Grafana) support annotations, which are markers overlaid on a time-series graph at a specific timestamp, commonly wired up to fire automatically whenever a deployment happens. This makes correlating a metric change with a recent release immediately visible to anyone glancing at the dashboard, instead of requiring someone to manually cross-reference two separate systems after the fact.

Question 6

Your p50 latency dashboard looks perfectly healthy, hovering around 80ms, but a growing number of customers are complaining about pages that take many seconds to load. The team initially dismisses the complaints because 'the dashboard shows we're fine.' What's the most likely explanation?

Accepted Answer

The p50 (median) only reflects the experience of the typical, fastest-behaving half of requests, and can look completely healthy even while a meaningful subset of requests sit far out in the slow tail; the team should be looking at a higher percentile like p95 or p99 instead. By definition, p50 (the median) only describes the point below which half of all requests fall, so it is mathematically insensitive to how bad the slowest half gets; a large chunk of requests could take 5+ seconds and the p50 would barely move. This is exactly why teams track higher percentiles like p95 or p99 for user-facing latency: they specifically expose that slow tail that the median, by design, is blind to, and dismissing real complaints because 'the median looks fine' is a common and costly mistake.

Question 7

In Prometheus's default monitoring model, how does the Prometheus server typically get metric data from an application?

Accepted Answer

Prometheus scrapes (pulls) metrics by making an HTTP GET request to an endpoint the application exposes, usually /metrics. Prometheus is a pull-based system: it periodically scrapes an HTTP endpoint (conventionally /metrics) exposed by the target application or an exporter sitting in front of it, and parses the plain-text metric format returned. This differs from push-based systems like StatsD, where the application itself sends metric updates outward. The pull model makes it easy for Prometheus to know a target is unreachable (a failed scrape) without needing separate heartbeat logic.

Question 8

A team is about to take a database offline for two hours of planned maintenance, which they know will trigger a wave of downstream alerts about failed connections. They don't want their on-call engineer paged for something they already know about and are already handling. What should they do before starting the maintenance?

Accepted Answer

Temporarily silence (mute) the specific alerts expected to fire because of the known maintenance window, then remove the silence once the work is done. A silence (or mute) suppresses notifications for specific, matching alerts during a defined time window, without deleting the underlying alert rule, so it automatically resumes normal behavior once the window ends. This is the standard way to avoid paging someone about a problem the team already knows is happening and intentionally caused, without weakening the alert's ability to catch a genuinely unexpected problem afterward.

Question 9

A team turns on an automated anomaly-detection alert for their traffic metric. It works fine on weekdays but pages the on-call engineer every single Saturday and Sunday morning, because traffic naturally drops on weekends and the tool flags that normal drop as anomalous. What's the most likely underlying cause?

Accepted Answer

The anomaly detection model isn't accounting for known seasonality (regular, predictable patterns like weekly or daily cycles) in the underlying data, and is instead comparing traffic against a baseline that assumes a flat, non-cyclical pattern. Good anomaly detection needs to model expected seasonality, the predictable, recurring patterns in a metric like lower traffic on weekends or at 3 AM, and compare current behavior against what's normal for that specific time context rather than a single flat baseline. Without that, the tool treats any deviation from an average as suspicious, which produces a steady stream of false positives for things that are actually completely normal, eventually training the on-call engineer to ignore the tool altogether.

Question 10

A developer is deciding what log level to use for a message that records 'user successfully completed checkout,' a normal, expected event that isn't an error but is still worth recording for business visibility. Which level fits best?

Accepted Answer

INFO. INFO is meant for notable, expected events in normal operation, things a team might want visibility into without them signaling a problem. DEBUG is for fine-grained detail useful only during active troubleshooting, and WARN/ERROR are reserved for situations that are unexpected or require attention, which a successful checkout is not.

Monitoring & Observability Quiz

1. You want to track the current number of items sitting in a queue, a value that goes up when items arrive and down as they're processed. Which metric type should you use?

2. An engineer is graphing the request rate of an HTTP counter metric using PromQL. Using irate() over a 5 minute range produces a jagged, spiky line, while rate() over the same range produces a smooth trend line. Why?

3. During an incident postmortem, a manager asks the writer to name the engineer who pushed the faulty config, so it's clear whose mistake caused the outage. What does a blameless postmortem culture say about this request, and why?

4. A page fires for 'disk usage above 90% on db-primary.' Attached to the alert is a link to a document with step-by-step instructions for how to safely free up space on that specific server. What is this document called?

6. Your p50 latency dashboard looks perfectly healthy, hovering around 80ms, but a growing number of customers are complaining about pages that take many seconds to load. The team initially dismisses the complaints because 'the dashboard shows we're fine.' What's the most likely explanation?

7. In Prometheus's default monitoring model, how does the Prometheus server typically get metric data from an application?

10. A developer is deciding what log level to use for a message that records 'user successfully completed checkout,' a normal, expected event that isn't an error but is still worth recording for business visibility. Which level fits best?

Ready for the real thing?

Welcome to OpsQuiz!