Distributed Tracing Sampling Strategy: How We Stopped Drowning in Spans

Head-based sampling at 1% misses the slow traces you care about—by definition, 99% of tail latency vanishes. Tail-based sampling keeps interesting traces after completion but needs buffering, memory, and collector topology you may not have planned for. We moved from head 1% to tail sampling on OpenTelemetry Collector 0.91.0 and cut span storage 85% while improving incident debuggability.

Previous state (head-based 1%)

• SDK: OTel Java 1.32, TraceIdRatioBased(0.01)
• Backend: Tempo 2.3, ~45M spans/day ingested
• Problem: checkout P99 spike to 4.2 s—grep found zero sampled traces above 2 s for affected window

Head sampling decides at trace start. Rare slow paths almost never enter the pipeline.

Tail-based approach

Collector tail_sampling processor buffers spans until trace completes (or timeout), then evaluates policies:

1. Always sample if http.status_code >= 500
2. Always sample if duration > 2s
3. Probabilistic 5% for remainder
4. Drop health checks and /metrics paths via ottl filter upstream

# collector fragment — otel-collector 0.91.0
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: slow
        type: latency
        latency: {threshold_ms: 2000}
      - name: baseline
        type: probabilistic
        probabilistic: {sampling_percentage: 5}

Operational costs:

• decision_wait: 10s delays trace visibility—acceptable for async analysis, painful for live dashboards
• num_traces: 100000 buffer ≈ 2 GB RAM on our collector pod during traffic spike; OOM once before we scaled
• Long-running batch jobs (20 min) need decision_wait tuning or forced flush spans—otherwise traces never close

Head vs tail tradeoffs

Aspect	Head-based	Tail-based
Collector memory	Low	High (buffer until complete)
Rare slow request capture	Poor	Good
Implementation location	SDK (simple)	Collector (centralized)
Late span arrival	N/A	Can miss decision if past wait

Instrumentation audit fallout

Tail sampling exposed garbage spans—SELECT 1 health checks labeled as DB spans, 400 k spans/day from misconfigured Redis client. OpenTelemetry adoption instrumentation audit cut noise before sampling policies could work.

Service mesh sidecar spans doubled trace depth; Envoy latency overhead meant parent spans looked slow while app spans looked fine—tail policy on total trace duration helped, but debugging required span attribute component=proxy filters.

Storage impact

• Before: 45M spans/day, $2.1k/month S3 + index
• After: 6.8M spans/day, similar useful trace count for incidents
• Effective sampling ~15% on interesting subset, ~0.3% on health noise

What I'd do next

1. Parent-based sampling in SDK for known high-volume workers—drop entire trace early if root is cron tick.
2. Dynamic tail policies via OTel opamp—raise slow threshold during deploy windows.
3. Trace quota per tenant—one customer’s fan-out graph should not evict buffer for others.

Collector sizing reference

Our production collector runs as a Kubernetes Deployment: 3 replicas, 4 vCPU / 8 GiB each after the OOM incident. Horizontal scaling helped ingest throughput but did not linearly scale tail buffer capacity—each replica buffers independently unless you shard by trace ID upstream. We added a hash ring on ingress gateway to pin trace streams to collector instances; that fixed duplicate tail decisions but introduced uneven load during single-tenant spikes.

For sizing estimates: assume average span size 500 B on wire, average trace depth 12 spans, buffer 100k traces → ~600 MB raw span payload plus Go runtime overhead. Add 40% headroom for attribute bloat when someone adds JSON blobs to spans (happened twice).

SDK vs collector sampling split

We kept 0.1% head sampling in SDK for ultra-high-volume cron paths that never error—reduces collector CPU before tail policy even runs. Application teams initially resisted ("my service disappeared"); documenting the two-layer model in the observability RFC helped. Rule of thumb: head sample noise, tail sample signal.

Tail sampling and long traces

Batch job traces exceeding decision_wait never closed—tail sampler dropped entire job trace. Fix: synthetic span batch.checkpoint every 5 min closes partial trace for sampling purposes; adds noise, preserves visibility.

Cost of retained attributes

One team attached base64 screenshot to error spans—storage 3× week over week until attribute size limit enforced in collector transform. OTTL drop rule: attributes["screenshot"] banned.

Tempo and retention coupling

Tail sampling increased interesting trace rate 5× but Tempo ingester CPU became bottleneck before storage—scaled ingesters horizontally before tuning sampling down. Sampling strategy and backend capacity are one design problem.

Exemplar traces for onboarding

New engineers replay stored exemplar traces in staging Tempo—slow checkout, payment error, mesh timeout—without production access. Tail sampling quality depends on someone curating teaching traces quarterly.

Service graph aware sampling

Downstream payment service errors triggered upstream checkout trace retention via tail policy linking trace IDs—requires consistent traceparent propagation validated in OpenTelemetry adoption instrumentation audit.

Stop drowning in spans: head sampling for volume services, tail policies for paths where milliseconds matter. Budget collector RAM like you budget Kafka retention—not as an afterthought.