OpenTelemetry Adoption: The Instrumentation Debt Audit We Should Have Done First

We migrated from three proprietary observability SDKs — Datadog APM agents, New Relic Java agent, and a homegrown StatsD wrapper — to OpenTelemetry 1.24 (SDK) with OTLP export to a Grafana Tempo + Mimir stack. The migration nearly took down Prometheus-compatible Mimir when cardinality exploded in week two.

We should have run an instrumentation debt audit before writing a single line of OTel code.

Pre-migration inventory (what we should have done first)

The audit we eventually ran — six weeks too late — cataloged:

Source	Spans/sec peak	Unique metric names	Cardinality offenders
Datadog Java agent	12k	auto-instrumented	http.url path params
Custom StatsD (Python)	3k	847	user_id tag on counters
NR Node agent	8k	auto + 120 custom	graphql.field per field

Total time series before migration: ~180k active (Mimir), within license.

After naive OTel port: 1.4M active series in 72 hours. Mimir ingesters OOM'd. On-call memorable.

What "naive port" looked like

Engineers translated existing patterns literally:

# Before (StatsD)
statsd.increment('checkout.started', tags=[f'user:{user_id}', f'merchant:{merchant_id}'])

# After (OTel — WRONG)
counter.add(1, {'user.id': user_id, 'merchant.id': merchant_id})

Perfectly reasonable copy-paste. Catastrophic for a metrics backend.

Same with traces: enabling OTEL_INSTRUMENTATION_HTTP_CAPTURE_HEADERS_SERVER=all because Datadog had captured headers for debugging — duplicated on 400 pods, Tempo storage bill jumped 34% in one month.

Instrumentation debt categories

We now classify every metric/span attribute:

1. Low cardinality — safe: service.name, http.method, db.system, deployment.environment
2. Bounded cardinality — OK with allowlist: http.route (templated), error.type (enum)
3. Unbounded — forbidden on metrics: user IDs, UUIDs, raw URLs, GraphQL field paths
4. Trace-only — OK on spans with tail sampling: request IDs for correlation when sampled

Documented in docs/observability-cardinality.md. CI linter (otel-lint, internal tool) rejects new metrics with forbidden attribute keys.

Migration phases that worked

Phase 0: Audit + budget (2 weeks)

• Export metric/span samples from each legacy system
• Compute cardinality per label key
• Set Mimir per-tenant series limit (hard cap at 250k, alert at 200k)
• Define sampling policy: head sample 10% errors always, 1% success on hot paths

Phase 1: Traces only, tail sampling

OTLP → OpenTelemetry Collector 0.96 → tail sampling processor → Tempo.

No custom metrics yet. Validated span volume, fixed service.namespace inconsistencies.

Phase 2: Metrics with translation table

Spreadsheet (yes, really) mapping old StatsD metrics → OTel names + allowed attributes. 847 → 312 metrics. Rest deleted as unused (nobody noticed for 30 days — debt confirmed).

Phase 3: Auto-instrumentation with overrides

Java: opentelemetry-java-instrumentation 2.1, disabled jdbc statement sanitization off, enabled http.route templating.

Node: @opentelemetry/auto-instrumentations-node 0.46, disabled fs instrumentation (noise).

Go: manual spans only on gRPC interceptors — auto-instrumentation still too immature for our grpc-go version mix.

Cardinality incident timeline

• Day 1: Mimir alert active series > 200k
• Day 2: Identified merchant.id on HTTP server duration histogram — 89k merchants
• Day 3: Emergency relabel drop in collector; lost 6 hours of merchant-segmented data (acceptable)
• Day 4: Audit retro; mandated Phase 0 for any team adding >5 custom metrics

Kafka + OTel interaction

Consumer lag metrics exported via OTel Prometheus exporter duplicated Kafka JMX exporter series. Two sources of truth, divergent values. Standardized on JMX → OTel receiver, removed custom lag gauges from app code. See Kafka partition strategy for the alerting side.

Collector pipeline topology

Final architecture after iteration:

App (OTLP gRPC) → Regional Collector (k8s DaemonSet)
  → batch processor (5s timeout, 8192 batch)
  → attributes/strip (drop forbidden labels)
  → tail_sampling (errors=always, default=probabilistic 10%)
  → exporters: Tempo (traces), Mimir (metrics), Loki (logs via filelog only)

Anti-pattern we removed: app pods exporting directly to Mimir remote_write — no central cardinality strip point, no sampling gate.

Collector CPU: 0.3 vCPU per node at peak on m6i.xlarge, acceptable. Memory spike during Mimir outage when queues backed up — added memory_limiter processor (512 MiB hard cap, spike to 768 MiB before drop).

Vendor SDK sunset checklist

Per legacy SDK decommission:

1. Confirm no dashboard panels reference old metric names (saved 12 orphaned Grafana panels)
2. Remove agent DaemonSet from node pools — Datadog agent was 0.15 vCPU/node forgotten cost
3. Update runbooks that said "check New Relic" — on-call muscle memory lags tooling

Total infra savings post-sunset: ~$4.2k/month vendor + ~18 vCPU cluster capacity recovered — justified migration project beyond observability unity alone.

What I'd do next

1. Metric RFC — any new metric name requires cardinality estimate: unique_values * scrape_interval
2. Weekly top-10 series report from Mimir /api/v1/status/tsdb — assign owners
3. Evaluate Prometheus native OTLP ingest when Mimir 2.12+ stable in our stack — one less collector hop

OpenTelemetry is not a drop-in replacement for vendor SDKs. It is a chance to delete instrumentation you should never have shipped — but only if you audit before you migrate, not after Mimir pages you at 3 AM.