PGA Stack - Operational Depth & Troubleshooting

Moving beyond basic dashboards to production-grade reliability and Alertmanager routing.

1. Alertmanager Configuration: Routing Trees & Inhibitions

Alertmanager receives alerts from Prometheus, groups related incidents, and silences noise.

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s        # Wait 30s to buffer initial alerts
  group_interval: 5m     # Send new alerts for the same group every 5m
  repeat_interval: 4h    # Resend identical alerts only every 4 hours
  receiver: 'slack-ops-notifications'
  routes:
    # Route critical alerts to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty-high-urgency'
      continue: true     # Also send to Slack

receivers:
  - name: 'slack-ops-notifications'
    slack_configs:
      - channel: '#prod-alerts'
        send_resolved: true

  - name: 'pagerduty-high-urgency'
    pagerduty_configs:
      - service_key: '<SECRET_KEY>'
        severity: critical

Inhibition Rules (Suppress Cascading Noise)

When an entire node dies, don't page 40 times for every container on that node:

inhibit_rules:
  - source_match:
      alertname: 'NodeDown'
    target_match:
      alertname: 'InstanceDown'
    equal: ['node', 'instance']

---

2. Kubernetes Prometheus Operator (ServiceMonitors & PrometheusRules)

In Kubernetes, configure scraping declaratively using ServiceMonitor CRDs:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-service-monitor
  namespace: payments
  labels:
    release: prometheus-stack
spec:
  selector:
    matchLabels:
      app: payment-api
  endpoints:
    - port: metrics
      path: /metrics
      interval: 15s