Back to Home
January 15, 20267 min read

Observability that on-call actually uses

Prometheus + Grafana SLIs: alerts humans can act on.

Problem

Dashboards looked great but didn't help during incidents because alerts lacked context and actionability.

Solution

Defined practical SLIs/SLOs, tuned Prometheus alerts, and created Grafana dashboards mapped to common failure modes.

Impact

On-call response improved with faster triage and fewer noisy, low-value alerts.

The biggest lie in DevOps is that more dashboards equal better observability. We had 40 different Grafana dashboards with hundreds of panels tracking CPU, Memory, I/O wait times, and thread counts. Yet, when PagerDuty went off at 3 AM, engineers were still paralyzed.

The Symptom vs. Cause Anti-pattern

Our alerts were configured based on system resources: "CPU Usage > 80% for 5m". This is fundamentally flawed because it alerts on the cause (resource saturation) rather than the symptom (customer experience degradation). Sometimes CPU runs hot during a perfectly healthy batch processing job.

Alert Fatigue

When 80% of your alerts resolve themselves or don't require action, on-call engineers learn to ignore them. When a real incident occurs, the critical alert gets lost in a sea of noise.

Transitioning to SLIs and SLOs

We threw away our resource-based alerts and adopted the RED method (Rate, Errors, Duration). We focused strictly on Service Level Indicators (SLIs) that directly correlated with user pain.

prometheus-rules.yml
- alert: HighErrorRate
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m])) 
    / 
    sum(rate(http_requests_total[5m])) > 0.05
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "High 5xx error rate detected"
    description: "More than 5% of requests are failing over the last 5 minutes."
    runbook: "https://wiki.internal/runbooks/high-error-rate"

Runbook Integration

Notice the runbook annotation in the Prometheus rule above. An alert without context is useless at 3 AM. Every critical alert we fire now requires a link to a maintained runbook that answers three questions:

  • What does this alert mean?
  • What is the immediate impact on the customer?
  • What are the first 3 diagnostic steps to isolate the issue?

The Result

Our overall alert volume dropped by 70%. When an engineer gets paged, they know exactly what the customer is experiencing and have a documented path forward. Triage time (MTTA) improved drastically.