Engineering Metrics That Lie: DORA, SPACE, and What I Measure Instead
DORA metrics are useful proxies. They become harmful when teams optimize the proxy rather than the outcome it was supposed to represent. What I watch instead.
DORA metrics — deployment frequency, lead time, change failure rate, MTTR — are useful proxies until teams optimize the proxy instead of the outcome. I've watched change failure rate improve while customer incidents worsened, because failures migrated to dark launches and "non-production" environments that still affected revenue.
This is a reading note on measurement practice, not a takedown of Forsgren et al. The research is fine. Our implementation was naive.
What DORA got right for us
- Deployment frequency correlated with smaller batch sizes when measured honestly (main branch deploys, not hotfix tags counted separately).
- Lead time for changes exposed review queue bottlenecks — our p95 sat at 9 days because of a single architecture review board meeting cadence.
- MTTR forced investment in runbooks and rollback automation we should have funded anyway.
These four metrics fit continuous delivery maturity conversations with the board. Keep them at that altitude.
Where DORA misled
Deployment frequency as vanity
Team A: 40 deploys/day of config toggles via LaunchDarkly counted as "deployments." Team B: 2 deploys/week of kernel patches. DORA dashboard crowned Team A "elite." Team B reduced CVE exposure. Wrong incentive.
Fix: Define deployment as artifact promotion (container digest change) with audit trail. Toggle-only changes excluded from headline metric; tracked separately.
Change failure rate without severity weighting
Rollback = failure. Hotfix revert = failure. P0 customer outage = failure (same 1 count). Teams learned to avoid rollbacks by leaving broken features flag-off for months — metric green, product dead.
Fix: Weight by incident severity × customer minutes. One P0 = 10 "failure points." Report both unweighted (for CD habit) and weighted (for honesty).
Lead time ends at deploy, not value
Median lead time 2 days — deploy to prod fast. Time to measurable user adoption of feature: 6 weeks (mobile app store, enterprise change windows). Optimizing lead time to deploy optimized the wrong loop for half our product surface.
MTTR gamed by redefining "restored"
"Service restored" = degraded mode with manual workaround. Customer still can't export data. MTTR 23 minutes on dashboard; support tickets open 4 days.
SPACE framework: same trap, softer numbers
Satisfaction, Performance, Activity, Communication, Efficiency — broader, harder to measure consistently. We ran a quarterly SPACE survey (15 questions, ~60% response). Results:
- Useful for qualitative trends ("communication score dropped after reorg")
- Useless for team comparison — self-reported, context-dependent
- Became another dashboard executives screenshot without reading methodology footnotes
SPACE isn't wrong for team health retros. It fails as org-wide KPI.
What I measure instead
Outcome metrics (customer-visible)
- Incident customer-minutes per month, tagged by root cause category
- SLO burn rate for tier-1 services (multi-window, multi-burn alerts)
- Support ticket escalation rate from engineering-caused defects (linked to Jira root cause field)
Flow metrics (honest)
- Batch size distribution — deploy count meaningless without diff stat histogram
- Review latency p50/p95 — separate from queue wait (time in "approved but not merged")
- Rollback count + partial rollback (flag disable without revert)
Learning metrics
- Postmortem action item closure rate at 30/60 days
- Repeat incident rate (same root cause category within 90 days)
- Blameless postmortem count is not a metric — it's a prerequisite
People metrics (private to managers)
- 1:1 themes, attrition risk signals, on-call load hours — never on public dashboards
- Tied to staff engineer scope transitions and hiring IC vs manager ratios — org design inputs, not leaderboard columns
Anti-patterns I refuse now
- Team rankings on DORA quadrants in all-hands slides
- Metric targets in performance reviews tied to deployment count
- SPACE survey scores compared across teams with different work types (infra vs feature)
Practical compromise
Public dashboard: SLO status, incident trends, deploy batch size histogram.
Leadership review (monthly): DORA four + weighted failure rate + lead time to prod (defined narrowly).
Team retro (biweekly): one SPACE dimension + one flow metric the team picks.
Case study: optimizing the wrong metric
Q2 2023: platform team celebrated "elite" deployment frequency after splitting monolith into 12 microservices — deploy count 3x'd. Change failure rate looked stable. Customer-reported defects on checkout rose 18% — same root bugs, smaller blast radius per deploy, more frequent partial rollouts without full integration test.
We were optimizing deploy count not integration confidence. Fix: mandatory canary analysis gate (Error Budget Policy) before full promote. Deploy frequency dropped; defect rate returned to baseline. DORA dashboard looked worse; customers noticed improvement.
Lesson embedded in our metrics dictionary under "deployment frequency — definition and anti-patterns."
What I'd do next
- Publish internal metrics dictionary — definition, owner, gaming risks, when to ignore
- Automate severity-weighted change failure from PagerDuty + deploy webhook correlation
- Stop buying tools that promise "DORA in a box" without definitional control
Measure outcomes customers feel, flows teams control, and learning loops that prevent repeats. DORA and SPACE are vocabulary, not destinations.
Manish Bookreader
Electronics enthusiast, Embedded Systems Expert, Linux/Networking programmer, and Software Engineer passionate about AI, electronics, books, and cooking.
