PGA Stack - Platform Engineering & Multi-Window SLO Burn Rates

1. Google SRE Multi-Window Multi-Burn-Rate Alerts

Traditional alerts based on static error rate thresholds fail:

• A 2% error rate might be safe for 1 minute, but catastrophic if sustained for 1 hour.
• A 100% error rate requires an immediate 3 AM page.

The 14.4x Burn Rate Alert (Consumes 2% of Error Budget in 1 Hour):

- alert: HighErrorRatePage
  expr: |
    (
      sum(rate(http_requests_total{status=~"5.."}[1h]))
      /
      sum(rate(http_requests_total[1h]))
    ) > (14.4 * (1 - 0.999))
    and
    (
      sum(rate(http_requests_total{status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total[5m]))
    ) > (14.4 * (1 - 0.999))
  for: 2m
  labels:
    severity: page
  annotations:
    summary: "14.4x burn rate detected on {{ $labels.service }} - 2% budget lost in 1h"

2. High Availability Alertmanager Mesh

Alertmanager uses a gossip protocol (HashiCorp Memberlist) to achieve zero-coordination HA:

• Deploy 3 Alertmanager replicas.
• Configure --cluster.peer flags to form a mesh.
• Replicas communicate notifications sent so a customer never receives duplicate pages even during network partitions.