Incident Response

Incident commander

The first goal is to mitigate, not to find the root cause. Keep messages short; the user is under pressure.

First 5 minutes

1. Impact: who is affected, what they can't do, since when. Ask only what is missing.
2. Severity (adapt to the company's scale if the user has one):
   • SEV1: core journey down for many users, money or data at risk.
   • SEV2: major degradation or a key feature down; workaround exists.
   • SEV3: minor impact, contained.
3. Roles: incident commander (coordinates, decides), ops lead (hands on keyboard), comms (updates). For small teams one person may hold two, never all three.
4. Open a channel and a timeline. Record every action with UTC-3 or UTC time, consistently.

Mitigation first

Ask "what changed?": deploys, config, feature flags, infra, traffic, dependencies. Prefer reversible actions in this order:

1. Roll back the last change or turn off the flag.
2. Shift traffic (failover region/AZ, drain bad nodes, weight away from the bad version).
3. Scale out or shed load (rate limit, disable non-critical features).
4. Only then, debug forward.

State the expected effect of each action and how you'll verify it (which graph should move).

Status updates

Every 30 min for SEV1, 60 min for SEV2, even if nothing changed. Template:

[SEV2] Payments slow for card authorisations — Investigating Impact: ~15% of card payments take over 10 s since 14:05. Doing now: rolling back release 4.12 of the auth service. Next update: 15:00.

No speculation about cause in external updates.

Closing

Declare mitigated when the SLI is back to normal for a sustained period. Then: write down the timeline while it's fresh, list follow-ups, and schedule the postmortem (use the postmortem skill).

Blameless postmortem

Principles

• Describe systems and decisions, not people's faults. "The deploy pipeline allowed X", not "João deployed X".
• Ask "how did it make sense at the time?" for every human decision.
• There is rarely a single root cause; list contributing factors.
• Action items must be specific, owned and dated. "Improve monitoring" is not an action item.

Gather

From what the user provides, extract: detection time, start time of impact, mitigation time, resolution time, the actions taken and who decided them. If times are missing, mark them as unknown instead of guessing.

Template

# <Title: what users experienced> — <date>

**Severity:** SEV<n>   **Duration of impact:** <hh:mm>   **Status:** draft

## Summary
Three sentences: what happened, impact, how it was resolved.

## Impact
Users/requests affected, business impact, SLO budget consumed.

## Timeline (UTC-3)
| Time | Event |
| --- | --- |

## Detection
How we found out. Time to detect. Did alerting work?

## Contributing factors
- Technical
- Process
- Detection/response

## What went well
## Where we got lucky

## Action items
| Action | Type (prevent / detect / mitigate) | Owner | Due |
| --- | --- | --- | --- |

Quality check before handing over

• Each contributing factor has at least one action item, or an explicit reason why not.
• At least one "detect" action if time to detect was long.
• The summary is readable by someone outside engineering.
• No blame language: search for "should have", "failed to", "forgot", "careless" and rewrite.

Key metrics to include when available: time to detect, time to mitigate, time to resolve.