Feature Flags in Production: The Operational Debt Nobody Budgets For

After eighteen months and three product releases, our feature flag system had 140 active flags and roughly three engineers who understood any given flag's interaction graph. Feature flags solve rollout risk. They create operational debt when removal is never prioritized. This note quantifies that debt and describes the cleanup system we eventually enforced.

Related: technical debt balance sheet framing, postmortem writing that sticks.

How we got to 140 flags

Legitimate origins:

• Gradual rollouts (new_checkout_flow_v2)
• Kill switches (payments_provider_failover)
• A/B experiments (onboarding_step_order_b)
• Entitlement gating (enterprise_sso_beta)
• Long-lived tenant overrides (customer_acme_legacy_api)

Illegitimate origins:

• Flags that should have been config (max_upload_size_50mb)
• Flags replaced by other flags (checkout_v2 + checkout_v2_1 + checkout_v2_hotfix)
• "Temporary" incident flags never removed (disable_cache_2023_03)

Nobody owned removal. Product assumed eng would clean up after launch. Eng assumed product would declare victory. Finance did not line-item flag storage on LaunchDarkly bill — $1,800/month seemed fine until we added HA SDK replicas.

Symptoms at 140 flags

Evaluation latency: SDK bootstrap payload 280 KB. Mobile cold start +400 ms p95 on 3G — traced to flag payload parse.

Test matrix explosion: CI ran combinatorial tests on "critical" flags — 2^6 combinations ignored, still 64 configs manually maintained.

On-call confusion: Incident runbook said "disable flag X." Flag X renamed in dashboard, code key still legacy_x, owner left company.

Wrong variant in prod: Flag defaulted true in code when SDK unreachable — safe for some flags, catastrophic for bypass_authz_check (yes, that name existed briefly — caught in review, story for another day).

Inventory exercise (week 1)

Exported LaunchDarkly API → CSV:

Field	Source
Key	API
Created date	API
Last modified	API
Last evaluated (prod)	API metric (if enabled)
Code references	ripgrep -r in monorepo
Owner	CODEOWNERS blame on first grep hit

Buckets:

1. Active — evaluated in 30 days
2. Zombie — not evaluated 90+ days but referenced in code
3. Orphan — not evaluated and no code reference
4. Kill switch — documented in runbook

Results: 140 total → 62 active, 41 zombie, 28 orphan, 9 kill switches.

Removal policy we wrote down

Every flag ticket requires:

## Flag lifecycle
- Owner: @handle
- Created: YYYY-MM-DD
- Remove by: YYYY-MM-DD (max 90 days for experiment, 180 for rollout)
- Default when removed: [true|false|config value X]
- Metrics proving safe removal: [link]

Remove-by date enforced in quarterly cleanup sprint — same ritual as dependency upgrades.

Kill switches exempt from 90-day rule but require annual review and postmortem link if ever toggled in prod.

Technical patterns for removable flags

Avoid:

if flags.get("new_flow"):
    new_flow()
else:
    old_flow()

Two codepaths forever.

Prefer:

NEW_FLOW = flags.get("new_flow", default=True)  # default matches post-removal world

def checkout():
    return new_flow() if NEW_FLOW else old_flow()

Better: single path with flag only gating exposure:

if flags.enabled("new_flow", user):
    return new_flow()
return old_flow()

After rollout: delete if, delete flag, one path.

Use config for non-binary values — upload limits belong in service config DB, not LaunchDarkly.

Cleanup sprint mechanics (week 3–4)

1. Delete 28 orphans from dashboard + verify no code (CI grep job)
2. For each zombie: owner must confirm remove or renew with written reason
3. Merge PRs removing code branches — flag deletion follows code, not precedes (avoid undefined flag defaults)

Removed 53 flags in one sprint. SDK payload 280 KB → 96 KB. Mobile cold start recovered 220 ms p95.

CI gates added

Flag reference check: every flags.get("key") must exist in flags/registry.yaml with owner and remove-by date. CI fails on drift.

Stale flag warning: nightly job flags keys where remove-by passed.

Default audit: script parses code for default values when SDK unavailable — security review for auth-related keys.

Cost framing for leadership

Used technical debt balance sheet language:

• Interest: 2 eng-days/quarter incident + test confusion ≈ $25k/year
• Direct cost: LaunchDarkly seats + SDK HA ≈ $22k/year (partially fixed by cleanup tier)
• Principal: 2-week cleanup sprint once, then policy

Easier sell than "flags feel messy."

When flags remain correct

• Real kill switches with runbooks (postmortem culture when used)
• Multi-tenant overrides with billing contract
• Short-lived experiments with pre-registered end dates

Not for: permanent business logic forks.

Metrics to track ongoing

Metric	Target
Active flags	< 60 for our org size
Flags past remove-by	0
Orphan flags	0
SDK bootstrap size	< 100 KB
Mean flag age	< 120 days

Dashboard in Datadog fed by nightly API scrape.

Postmortem tie-in

When incident mitigated by toggling flag, postmortem action item defaults to:

1. Document kill switch in runbook
2. Schedule removal or promote to config with owner

See postmortem writing that sticks.

What I would do differently

Enforce registry.yaml from flag #1 — retrofitting at 140 hurt.

Charge flag creation to product team's capacity budget — same as API surface expansion.

Never name a flag disable_security_* — naming review for auth flags.

Related

Technical debt framing for executive conversations. Postmortems when kill switches fire.