The Prometheus, Grafana & Alertmanager Stack for Docker and Kubernetes Monitoring

Ask five different teams what their "monitoring stack" looks like and most will describe some variation of the same three tools: Prometheus, Grafana, and Alertmanager. Each does one specific job, and understanding the boundary between them is the key to understanding how the whole thing fits together.

Prometheus: the one that collects and stores metrics

Prometheus is a time-series database purpose-built for monitoring. Unlike a traditional monitoring agent that pushes data to a central server, Prometheus works the other way around: it pulls (scrapes) metrics from targets at regular intervals, over plain HTTP, from a /metrics endpoint that the target application exposes.

# prometheus.yml
scrape_configs:
  - job_name: 'my-app'
    scrape_interval: 15s
    static_configs:
      - targets: ['my-app:8080']

On Kubernetes specifically, Prometheus doesn't need a static list of targets, it uses service discovery to automatically find Pods and Services to scrape based on annotations, which is essential in an environment where Pods come and go constantly and a static target list would be permanently out of date.

Metrics come in a handful of core types: counters (a value that only goes up, like total requests served), gauges (a value that goes up and down, like current memory usage), and histograms (a distribution of observed values, like request latency buckets). Choosing the right type matters, since it determines what kind of query makes sense against that metric later.

Querying is done with PromQL, Prometheus's own query language:

rate(http_requests_total[5m])

That query alone answers "what's the average per-second request rate over the last 5 minutes", using a counter metric that just accumulates upward on its own.

Alertmanager: the one that decides who gets paged, and when

Prometheus can evaluate alerting rules and fire alerts, but it deliberately doesn't handle what happens next, that's Alertmanager's job, and the separation is intentional. Alertmanager receives fired alerts and handles:

• Grouping: bundling related alerts (say, twenty Pods all failing the same health check) into a single notification instead of twenty separate pages.
• Deduplication: if the same alert keeps firing every evaluation cycle, you get one notification, not one every few seconds.
• Routing: sending different alerts to different places, a database alert to the DBA team's Slack channel, an infra alert to PagerDuty for the on-call SRE.
• Silencing: temporarily muting a known issue (during planned maintenance, for example) without having to disable the underlying alert rule.

# alertmanager.yml
route:
  receiver: 'default-slack'
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty-oncall'

receivers:
  - name: 'default-slack'
    slack_configs:
      - channel: '#alerts'
  - name: 'pagerduty-oncall'
    pagerduty_configs:
      - service_key: '<key>'

This is the piece that turns "a metric crossed a threshold" into "the right human got notified through the right channel, without being spammed."

Grafana: the one you actually look at

Prometheus has a basic built-in expression browser, but it's not built for building real dashboards. Grafana connects to Prometheus (and many other data sources) as a backend and turns PromQL queries into the graphs, tables, and dashboards people actually look at during an incident or a weekly review.

Grafana's real value beyond "it makes pretty graphs" is in how it lets you compose multiple metrics into a single view that tells a story, request rate next to error rate next to latency percentiles, so you can visually correlate a spike in errors with whatever else was happening at the same moment, rather than mentally cross-referencing three separate tools.

Grafana can also fire its own alerts independently of Alertmanager, which is worth knowing so you don't end up with two overlapping, uncoordinated alerting systems by accident. Most teams pick one place to own alerting logic, usually Prometheus and Alertmanager, and use Grafana purely for visualization.

How the three actually connect

Your app  --exposes metrics-->  Prometheus  --evaluates alert rules-->  Alertmanager --notifies--> Slack/PagerDuty
                                     |
                                     +--queried by-->  Grafana  --displayed as-->  Dashboards

Prometheus is the source of truth for the data. Alertmanager decides what to do when something's wrong. Grafana is how a human actually looks at any of it. None of the three replaces either of the others, they're deliberately narrow in scope, which is exactly why the combination has stayed the default choice for so long instead of being replaced by an all-in-one tool.

A common beginner mistake

Setting up Prometheus and Grafana but skipping Alertmanager entirely, and instead configuring alert notifications directly inside Grafana dashboards. It works for a small setup, but it means your alerting logic lives scattered across individual dashboards rather than in one place you can reason about, test, and version-control as a whole. Starting with Alertmanager from day one, even for a small setup, avoids having to untangle this later.

Once you're comfortable with how metrics, alerting, and visualization divide up, the Monitoring & Observability quiz on OpsQuiz is a solid way to check how well it's actually stuck.