Incident Response at 2 AM: A Runbook for Engineering Leads
The worst incidents I've been part of were slow not because the fix was hard but because decision authority was unclear. A runbook that helps with that.
The worst incidents I've been part of were slow not because the fix was hard but because decision authority was unclear. A runbook for engineering leads when pages fire at 2 AM — distilled from six production incidents over four years (SaaS + fleet firmware).
Answer first
Your job in the first 15 minutes isn't fixing — it's stabilizing communication, assigning roles, and stopping parallel untracked changes. ICs debug; you coordinate, protect the ICs from distraction, and decide rollback vs. forward fix with evidence.
Roles (explicit, out loud)
| Role | Who | Does |
|---|---|---|
| Incident Commander (IC) | On-call lead or you if paged as eng lead | Timeline, decisions, external comms |
| Tech Lead | Strongest domain owner present | Hypothesis, fix direction |
| Scribe | Anyone literate | Timestamped log in shared doc |
| Comms | PM or you if solo | Status page, customer-facing |
| Subject experts | DB, embedded, net | Called in; not all in war room |
If five people are all "helping debug" with no IC — you already lost 20 minutes.
First 15 minutes checklist
[ ] Acknowledge page — "I'm IC" in Slack/incident channel
[ ] Open incident doc from template (link pinned in #incidents)
[ ] Ask: customer impact? SLO breach? data loss risk?
[ ] Freeze non-essential deploys
[ ] Identify last known good change (deploy, config, fleet OTA)
[ ] Assign scribe NOW — not after "we'll remember"
Don't solve yet. Stabilize who decides.
Decision framework: rollback vs. forward
Rollback when
- Clear correlation to deploy/config change < 2 h ago
- Fix unknown or requires > 30 min investigation
- Data integrity not yet compromised
- Rollback tested path exists
Forward fix when
- Rollback unavailable (schema migration one-way, OTA slot already swapped)
- Rollback worse (old version has active CVE exploited now)
- Fix is one-line feature flag with known safe default
Default: rollback if unsure. Forward fixes at 2 AM have higher defect rate — we've measured it twice informally (~2× re-open rate).
Document decision in incident log with one sentence why.
Communication templates
Internal (every 30 min until stable)
Status: investigating | mitigating | monitoring
Impact: [who/what]
Current theory: [one line]
Next update: [time]
Needs: [specific ask — not "help"]
External (via PM/comms)
No root cause until you know. "We are investigating elevated error rates on API X" beats silence. Silence breeds executive Slack threads — worse than partial truth.
Protecting ICs
Engineering lead duty:
- Shield debugger from "any updates?" every 4 minutes — you answer stakeholders
- Kill parallel SSH prod tweaks — one change pipeline; scribe logs who ran what
- Enforce break at 90 minutes — tired people ship bad SQL
On-call rotation design matters — see sustainable on-call. If same person IC + sole debugger third night running, escalate hire or rotation fix after incident.
Technical patterns by domain
Database (Postgres)
Check replication lag, connection saturation, long queries — our index bloat incident looked like app bug until pg_stat_activity showed autovacuum blocked. Don't restart primary without understanding — vacuum tuning prevents repeat.
Fleet firmware
Stop OTA campaign first — MCUboot rollback path may save bricked subset; correlate failure geography with charger type if USB-PD related.
Cloud cost spike
Sudden egress — honest accounting playbook — often CDN misconfig not attack. IC assigns one person to billing console, not whole team refreshing Cost Explorer.
Escalation triggers
Page exec when:
- Data loss confirmed or likely
- SLA breach > 1 h customer-visible
- Legal/compliance (PII exposure)
- Need business decision on rollback that kills quarterly feature
Not when: "it's scary" but customers fine and error rate 0.1% above baseline.
After mitigation
- Monitoring hold — 30–60 min green dashboards before "resolved"
- Preliminary timeline in doc within 4 h — scribe promotes rough notes
- Schedule postmortem within 3 business days
- File tickets for follow-ups with owners before closing incident channel
Anti-patterns I've seen
- Hero debugging — one senior fixes alone, no log; knowledge evaporates
- Blame question first — "who deployed?" before "what's impact?" — hides problems
- Permanent incident channel — becomes chat; archive and new channel per incident
- Skip postmortem because "we fixed it fast" — repeats in 6 months (we did, twice)
Runbook artifacts (repo)
We keep in docs/incidents/:
INCIDENT_TEMPLATE.md— roles, checklist, commsROLLBACK.md— per-service stepsCONTACTS.md— vendor TAM, AWS support case template
Print QR to template in ops wiki — searching Notion at 2 AM fails.
What I'd do next
Quarterly game day — inject failure, rotate who plays IC. Last game day exposed our scribe doc permission bug (ironic).
Metric: time-to-role-assignment — target < 5 min from page. We improved from 18 min median to 4 min over 6 months by literally reading this checklist aloud on first ping.
Incident response is a management protocol, not a technical skill. The fix might be one SQL statement; the cost is decided by how you run the room.
War room tooling
- Zoom/Meet with single link pinned — not new link per escalation
- Google Doc scribe template with read-only share to exec observers — reduces Slack DM noise
- PagerDuty incident channel auto-post to Slack — single
#incident-YYYYMMDD-hhmm
Ban parallel WhatsApp group — information silo guaranteed.
Status page policy
Update within 20 min of customer-visible SEV1/2. Yellow "monitoring" only after error rate < baseline × 1.1 for 15 min — not immediately after deploy fix.
Post-incident manager duties
Schedule postmortem before closing PagerDuty incident — calendar invite in same breath as "mitigated."
What I'd do next
Manish Bookreader
Electronics enthusiast, Embedded Systems Expert, Linux/Networking programmer, and Software Engineer passionate about AI, electronics, books, and cooking.
