Observability & SLOs

Alert review

Goal: every page is urgent, actionable and needs a human. Everything else is a ticket, a dashboard or deleted.

Inputs

Read the alert definitions in the repo. If available, also get firing history (Alertmanager/PagerDuty/Opsgenie export) for the last 30 days. Without history, review definitions only and say so.

For each alert, decide

Symptom alerts (error rate, latency, SLO burn) should page. Cause alerts (CPU high, pod restarted, disk 80%) usually should not, except saturation that will cause an outage soon (disk full in < 4h via predict_linear).

Alertmanager hygiene

• Grouping by alertname, cluster, namespace/service so one incident is one notification.
• Inhibition: cluster or node down inhibits the per-pod alerts below it.
• Routing: severity page to the on-call tool, ticket to a channel or queue, never both.
• Every alert has summary, description with the values, runbook_url and an owner label.

Output

1. Summary numbers: total alerts, pages, proposed removals/downgrades.
2. A table: alert, current severity, verdict (keep / tune / downgrade / delete / merge), reason.
3. Missing coverage: user journeys with no symptom alert.
4. Patches for the changes.

PromQL helper

Writing queries

• Find the metric type first. Counters (_total) always go through rate()/increase(); never sum() a raw counter.
• rate() before sum(): sum by (svc) (rate(x_total[5m])), never rate(sum(x_total)[5m]).
• The range should be at least 4× the scrape interval (with 30s scrapes, use [2m] or more).
• Percentiles from histograms keep the le label:

histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m])))

Native histograms: histogram_quantile(0.99, sum by (route) (rate(http_request_duration_seconds[5m]))).

• Ratios: aggregate numerator and denominator with the same by clause.
• Joins: use on()/ignoring() with group_left to pull labels from info metrics like kube_pod_info.
• Absent data: absent() or absent_over_time() for "is the target dead" alerts.

Review checklist

1. Counter used without rate? Gauge used with rate?
2. Aggregation drops a label the alert or dashboard needs?
3. histogram_quantile without le in the by? Averaging percentiles across instances (wrong)?
4. Regex matchers on high-cardinality labels in hot dashboards?
5. Range too short for the scrape interval?
6. Should this be a recording rule (used in several panels/alerts, or expensive)?

Cardinality

When the user suspects a cardinality problem:

topk(10, count by (__name__) ({__name__=~".+"}))
topk(10, count by (le) (http_request_duration_seconds_bucket))

Look for labels carrying user ids, full URLs, request ids or timestamps. Recommend dropping them with metric_relabel_configs or fixing the instrumentation.

Always explain what the final query returns in one sentence, including the unit.

SLO design

1. Start from the user journey

Ask (or infer from the code/config) what the service's users actually do: e.g. "authorize a payment", "load the statement page". Pick 1–3 journeys. An SLO per journey beats an SLO per microservice.

2. Choose SLIs

Express every SLI as good events / valid events:

Measure as close to the user as possible (load balancer or mesh beats app-internal). Exclude 4xx from "bad" unless the service caused them.

3. Set the objective

• Base it on what users tolerate and on historical performance, not on "five nines".
• Window: 28 or 30 days rolling.
• Error budget = 1 − SLO. At 99.9% over 30 days that is ~43 minutes of full outage.
• Write down the error-budget policy: what happens when it is spent (feature freeze, reliability work prioritised).

4. Alerts: multi-window, multi-burn-rate

Use the standard pairs (for a 30-day window):

Template (availability, SLO 99.9%):

groups:
- name: slo-checkout
  rules:
  - record: slo:sli_error:ratio_rate5m
    expr: |
      sum(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))
      / sum(rate(http_requests_total{job="checkout"}[5m]))
  # repeat the recording rule for 30m, 1h, 2h, 6h, 1d, 3d
  - alert: CheckoutErrorBudgetBurnFast
    expr: |
      slo:sli_error:ratio_rate1h > (14.4 * 0.001)
      and slo:sli_error:ratio_rate5m > (14.4 * 0.001)
    labels: {severity: page}
    annotations:
      summary: "Checkout is burning its error budget 14x too fast"
      runbook_url: "<link>"

If the stack uses Sloth or Pyrra, generate their spec instead of raw rules.

5. Output

Deliver: the journeys, SLI definitions with the exact query, the objective and its justification, the alert rules, and a dashboard outline (SLI over time, budget remaining, burn rate). Point out SLIs that can't be measured yet and what instrumentation is missing.