Postmortem Writing That Actually Sticks

Most postmortems get filed and forgotten. The difference between a useful one and a formality is specificity of contributing factors and follow-up ownership — we changed our template after postmortems #4 and #7 still produced repeat incidents.

Answer first

A postmortem that sticks names multiple contributing factors (not one root cause hero), lists detection gaps, and assigns owned follow-ups with due dates tracked in the same system as product work — not a Google Doc graveyard.

Bad vs. good (same incident)

Bad (actual excerpt, redacted)

Root cause: human error during deploy.
Action items: be more careful; add checklist.

Filed. Three months later: same wrong migration order, different human.

Good (rewrite of same event)

Impact: 47 min API 503, 12% of requests, ~$8k SLA credits
Contributing factors:

1. Migration 20240315 ran before app version expecting column (ordering doc unclear)
2. Staging lacks prod-scale data — migration duration underestimated 40×
3. Rollback script untested since Q2
   Detection gap: 11 min until alert; no synthetic check on /health/deep
   Follow-ups:

• [ENG-4412] CI test rollback on clone @alice due Apr 1
• [ENG-4413] Expand staging DB snapshot weekly @bob due Apr 8
• [ENG-4414] Deploy runbook migration order section @carol due Mar 20

Two follow-ups shipped; third incident class eliminated.

Template structure we use

1. Summary — 3 sentences max, exec-readable
2. Impact — duration, users, revenue, data — numbers
3. Timeline — UTC timestamps, link to incident doc
4. Contributing factors — numbered; forbid single "root cause" line without factors
5. What went well — builds blameless culture for real
6. Detection / response gaps
7. Follow-up actions — ticket ID, owner, date
8. Appendix — graphs, queries, log snippets

No "five whys" mandate — often performsative. Ask "what conditions made this possible?" instead.

Blameless vs. accountability

Blameless ≠ nameless. We name systems and decisions, not "Bob is careless."

OK: "On-call runbook didn't list rollback command; Bob reasonably tried restart first per outdated wiki."

Not OK: "Bob restarted without thinking."

If someone must be coached, manager private — not postmortem audience.

Ties to incident response runbook: IC ensures postmortem scheduled before heroes disperse.

Contributing factors taxonomy

Helps teams stop at first answer:

Category	Example
Process	migration order undocumented
Technical debt	rollback script bit-rotted
Tooling	CI missing integration test
Observability	alert threshold too lazy
Capacity	Postgres bloat unknown until stall
External	vendor API degraded

Aim for ≥3 categories represented when severity ≥ SEV2. If all factors are "process," you're probably protecting a design flaw.

Follow-ups that die (and fixes)

Problem: action items live in postmortem PDF.

Fix: every item → Jira/Linear ticket same day; incident channel pinned until all closed or explicitly wont-fix with reason.

Problem: "monitor more closely."

Fix: require metric or alert name in ticket — "add alert api_p99 > 2s for 5m page on-call."

Problem: no priority.

Fix: SEV1 postmortem items default P1; block next sprint planning until P1s assigned.

Review open postmortem tickets in eng weekly — 5 min standing agenda.

Audience-specific exports

• Exec summary: impact + prevention headline only
• Eng deep dive: full doc
• Support: customer-facing language, no internal tool names

Same incident, three extracts — auto from headings, not rewrite marathon.

Postmortem quality rubric (peer review)

Score 1–5; publish average (anonymous) to eng:

1. Timeline has minute granularity
2. ≥2 contributing factors with evidence
3. Every action has owner + date + ticket
4. Detection gap addressed
5. Linked to prior related incidents if any

Below 3 average → staff engineer helps rewrite — not punishment, coaching.

Related incidents linking

Tag incident-theme:deploy-migration in doc. Search before closing — if third migration incident, escalate to design change (expandable/contract migrations mandatory), not fourth checklist.

Cross-link on-call rotation when fatigue factor — "primary on-call 3rd consecutive night" belongs in factors, not excuse.

When NOT to write full postmortem

• SEV3 self-inflicted dev-only < 5 min — lightweight note OK
• Duplicate of open postmortem theme with same fix in flight — append to existing

Always write for data loss, customer visible SEV1/2, or missed SLO.

What changed after we got serious

• Repeat incident rate (same theme tag) down ~60% YoY — not zero, honest
• Mean time to postmortem draft: 36 h → 14 h
• Action item completion at 30 days: 40% → 78%

What I'd do next

Automate timeline import from PagerDuty + Slack export → incident doc headings. Scribes spend time on factors, not copy-paste UTC.

Quarterly postmortem reading club — one doc, eng discussion; better than policy emails.

Postmortems stick when they're project plans in disguise — same accountability as features, same tracking, same respect.

Example contributing factors (anonymized SEV1)

Database failover incident:

1. Primary disk hit IOPS ceiling — autovacuum + analytics query overlap
2. Replica promotion runbook referenced PG13 syntax — cluster PG14
3. Failover DNS TTL 300 s — apps cached old IP 4 min
4. On-call primary hadn't run failover drill in 8 months

No single "root cause" — four fixes, four owners, zero repeat in 12 months.

Wont-fix criteria

Some actions legitimately closed wont-fix with executive sign-off — document why. "Accept risk: cross-AZ cost vs 99.95% SLA sufficient" — better than zombie ticket forever.

What I'd do next