Monitoring & Observability Quiz

A counter is a cumulative metric whose value only ever goes up, or drops back to zero when the process restarts; it's the right choice for totals like requests served or errors raised. A gauge, by contrast, can go up or down (like current memory usage), which is why counters and gauges are treated differently by query functions like rate().

When one database goes down, every service that depends on it starts failing too, and each of those failures fires its own alert - so you get flooded with pages that are really all symptoms of the same root problem. Alert correlation groups related alerts together and only pages you for the root cause, so on-call gets one clear signal instead of a hundred confusing ones.

The whole point of an error budget is to make it an actionable policy: once it's mostly or fully spent, standard practice is to halt risky changes, like new feature launches, and redirect engineering effort toward reliability until the budget replenishes over the rolling window. Quietly lowering the SLO to make the numbers look better defeats the purpose of having an objective in the first place, and treating the budget as retroactive-only ignores that its main value is as a forward-looking decision trigger. There's also no such thing as 'rolling back to the SLA,' an SLA is a separate contractual layer, not something you revert to.

A liveness probe answers 'is this process stuck or deadlocked?', and failing it typically triggers a restart. A readiness probe answers 'can this instance currently serve traffic?', such as during startup while it's still warming a cache or waiting on a dependency, and failing it just removes the instance from load balancing without restarting it. Conflating the two is a common mistake that causes unnecessary restarts of instances that just need more time to warm up.

A normal alert only tells you something when a bad condition is detected, which is useless if the alerting system itself is the thing that's broken. A dead man's switch inverts this: it's an alert (or external check) configured to expect a regular 'I'm still alive' heartbeat, and it's the missing heartbeat itself that triggers the page, which is why it can catch failures in the monitoring pipeline that would otherwise monitor itself into silence.

Infrastructure metrics like CPU and request rate describe the health of the machinery, but they can't tell you whether the actual business logic running on top of it is succeeding, a broken step deep in the signup flow could still return valid-looking HTTP 200s and healthy request rates. Custom application metrics, emitted directly from application code to track specific business events, are what makes this kind of silent functional regression visible, and it's a common blind spot for teams who only instrument generic infrastructure signals.

The four golden signals are latency, traffic, errors, and saturation, and saturation specifically measures how close a system is to its resource limits (CPU, memory, queue depth, and so on) before performance degrades. The other options describe engineering process or delivery metrics, not signals you'd track on a service's core health dashboard.

Many dashboarding tools (like Grafana) support annotations, which are markers overlaid on a time-series graph at a specific timestamp, commonly wired up to fire automatically whenever a deployment happens. This makes correlating a metric change with a recent release immediately visible to anyone glancing at the dashboard, instead of requiring someone to manually cross-reference two separate systems after the fact.

Dashboard variables let a single dashboard definition stay generic, with panel queries referencing a variable like $service instead of a hardcoded service name, and a dropdown at the top lets whoever's viewing it pick which specific value to plug in. This turns what would otherwise require 40 separate, hand-maintained dashboards into one reusable template, which is exactly the maintenance burden the scenario describes.

An SLO (Service Level Objective) is the target you set for an SLI, and 'the objective of 99.9% of requests under 300ms' is exactly that target built on top of the underlying SLI (the raw measured percentage of fast requests). The SLI alone would just be 'the percentage of requests under 300ms,' with no target attached; an SLA layers a customer-facing contractual consequence on top of an SLO, which this statement doesn't mention.