PGA Stack - Production Scaling & Reliability

How to run Prometheus and Alertmanager reliably at scale without OOM crashes or missing critical outages.

1. Preventing High-Cardinality Outages

The #1 cause of Prometheus crashes is High Cardinality - metrics with label values that have millions of unique combinations (e.g., user_id, order_id, ip_address).

Identifying High Cardinality:

In Prometheus web UI: Navigate to Status -> TSDB Status to inspect:

• Top 10 label names with highest cardinality
• Top 10 series count per metric name

Relabeling to Drop Dangerous Labels:

scrape_configs:
  - job_name: 'api-service'
    metric_relabel_configs:
      - action: labeldrop
        regex: '(user_id|trace_id|session_token)'

---

2. Scaling Architecture: Thanos & Prometheus Agent Mode

A single Prometheus instance cannot scale indefinitely. When series exceed 5 million:

1. Prometheus in Agent Mode: Runs lightweight Prometheus instances that only scrape and stream metrics via Remote Write to object storage.
2. Thanos Sidecar / Receiver:
   • Ships 2-hour TSDB blocks to AWS S3 / Google Cloud Storage.
   • Thanos Query: Provides a global deduplicated view across multiple clusters.
   • Thanos Compact: Downsamples historical data (5m resolution after 40 hours, 1h resolution after 2 weeks).