Prometheus, Grafana & Alertmanager (PGA Stack) - Foundations

The PGA Stack (Prometheus, Grafana, Alertmanager) is the industry standard for cloud-native observability and incident alerting.

1. Core Architecture Overview

┌─────────────────┐       Scrapes /metrics        ┌──────────────┐
│  Microservices  │ ◄──────────────────────────── │  Prometheus  │
│  & NodeExporter │                               │    Server    │
└─────────────────┘                               └──────┬───────┘
                                                         │
                         Queries (PromQL)                │ Evaluates AlertRules
                         ┌───────────────────────────────┴───────────────┐
                         ▼                                               ▼
                 ┌──────────────┐                               ┌─────────────────┐
                 │   Grafana    │                               │  Alertmanager   │
                 │ (Dashboards) │                               │  (Routing & Pager)
                 └──────────────┘                               └────────┬────────┘
                                                                         │
                                                                         ▼
                                                           [Slack / PagerDuty / Email]

• Prometheus: Time-series database that pulls (scrapes) metrics over HTTP at configured intervals.
• Grafana: Visualization layer that queries Prometheus using PromQL and renders rich dashboards.
• Alertmanager: Deduplicates, groups, and routes alert notifications fired by Prometheus to Slack, PagerDuty, or webhooks.

---

2. The 4 Fundamental Metric Types

1. Counter

A cumulative metric that only increases (or resets to 0 on restart).

• Use cases: Total HTTP requests, completed jobs, error count.
• PromQL rule: Always use with rate() or increase(). Never graph raw counters.

# Request rate per second over 5 minutes
rate(http_requests_total[5m])

2. Gauge

A metric that can go up or down.

• Use cases: Memory usage, active TCP connections, CPU temperature, queue depth.

# Current node memory in bytes
node_memory_MemAvailable_bytes

3. Histogram

Samples observations (usually durations or response sizes) and counts them into configurable buckets.

• Use cases: Request duration (p95, p99 latencies), payload sizes.

# Calculate 95th percentile request latency over 5 minutes
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

4. Summary

Calculates streaming quantiles directly on the client side.

• Note: Cannot be aggregated across multiple pods or servers. Histograms are preferred in cloud-native environments.

---

3. Writing Your First PromQL Queries

1. Filtering by Labels:

http_requests_total{status="500", job="api-gateway"}

2. Calculating Error Rate:

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

3. Disk Space Prediction (Predict linear saturation in 4 hours):

predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0