Service Mesh Overhead: Measuring Envoy Proxy Latency on Our Workload

We adopted Istio 1.20 on Kubernetes 1.28 (EKS) for mTLS, traffic policy, and because the platform team wanted unified observability without instrumenting every service. The question that survived the POC: how much latency does the Envoy sidecar actually add on our workload mix?

Answer first: ~1.2 ms p50, ~4.1 ms p99 east-west on identical RPC paths, measured with and without sidecar injection. Acceptable for most services. Unacceptable for two latency-sensitive paths we later exempted.

Measurement setup

Workload: Go 1.22 gRPC services, average payload 2.4 KB protobuf, 1.2 KB response. Mix of 60% unary, 40% streaming (small batches). Node type: m6i.xlarge, Cilium CNI (not Istio CNI — we use istio-cni plugin v1.20).

Method:

1. Baseline: sidecar.istio.io/inject: "false" on namespace payments-baseline
2. Meshed: same deployment in payments-meshed with injection enabled
3. Traffic: fortio + in-house load gen at 2k RPS per service pair
4. Metrics: Prometheus histogram istio_request_duration_milliseconds vs app-native grpc_server_handling_seconds

We also ran kubectl exec curl tests service-to-service for sanity — not for p99 claims.

Envoy version: bundled with Istio 1.20.2, Envoy proxy ~1.28.x.

Results by path type

Path	p50 overhead	p99 overhead	Notes
Same-AZ unary gRPC	0.8 ms	2.9 ms	Best case
Cross-AZ unary	1.4 ms	4.1 ms	AZ boundary dominates tail
Small streaming (10 msgs)	1.9 ms	6.2 ms	Buffering in sidecar
Large payload (50 KB)	2.1 ms	8.7 ms	Memory copy cost

CPU overhead: ~0.08 vCPU per sidecar at 500 RPS, spiking to 0.15 vCPU at 2k RPS per pod. Memory: ~80 MB resident per sidecar (limit 256 MB).

What drove the tail latency

Not TLS — PERMISSIVE mTLS then STRICT showed negligible delta (<0.1 ms p50) once session caches warmed.

Actual contributors:

1. Extra hop + userspace proxy. Even fast Envoy adds context switches.
2. connect_timeout and route complexity. We had 47 VirtualServices; trimmed to 12 with shared templates.
3. Access logging. JSON access logs to stdout at info added ~0.6 ms p99. Disabled on hot paths; use telemetry v2 metrics instead.
4. CPU throttling on sidecar. Sidecar limit 256Mi / 0.2 CPU was too tight; bumped to 0.5 CPU on payment hot paths.

Services we exempted

• Fraud scoring inference gateway — p99 SLA 15 ms end-to-end; 4 ms proxy tax was 27% of budget. Runs in dedicated namespace, inject: false, mTLS via Cilium instead.
• Real-time quote fanout — WebSocket-ish long poll; sidecar connection idle timeouts caused spurious 503s until we tuned idleTimeout.

Everything else stays meshed. The exemptions are documented in ADR-0142 with re-review quarterly.

mTLS overhead in isolation

We A/B tested PERMISSIVE vs STRICT mTLS on the payments path — 10M requests sample:

Mode	p50 delta vs plain	p99 delta
PERMISSIVE (auto-migrate)	+0.05 ms	+0.2 ms
STRICT	+0.08 ms	+0.3 ms

Session resumption mattered: first request after pod start +1.2 ms p99; steady state negligible. Cold start budget must include mesh handshake if you enforce STRICT on day one.

Certificate rotation via Istio CA (24h cert lifetime): zero-downtime rotation observed, but Envoy hot restart during cert push caused 0.01% 503 spike once — tracked to overly aggressive terminationDrainDuration.

Resource accounting for platform teams

Sidecar tax isn't only latency. At 400 meshed pods:

• +32 vCPU cluster-wide at average load (0.08 vCPU × 400)
• +32 GB RAM requested (80 MB × 400)

We almost missed a cluster autoscaler trigger — memory requests bumped node count before CPU saturated. Mesh adoption should be in capacity planning spreadsheets, not only architecture diagrams.

Comparison to alternatives considered

• Linkerd 2.14: Tested on one cluster; p50 ~0.5 ms lower on same workload, smaller proxy. Chose Istio for team familiarity and Gateway API support — org decision, not pure perf.
• Cilium service mesh (beta): eBPF path promising; not GA for our EKS version when we decided.

See Linux cgroups and container memory limits for how sidecar memory limits interact with OOM kills — separate incident, same namespace.

What I'd measure if starting over

1. Production shadow traffic before mandating injection org-wide — our lab RPS was optimistic.
2. Tail by payload size decile — averages hide 50 KB outliers.
3. Cold start / first request after pod rollout — connection pool warmup added 40 ms on first 100 requests post-deploy.

What I'd do next

• Evaluate ambient mesh (Istio 1.22+ sidecar-less mode) on staging — sidecar tax should drop if eBPF path matures.
• Consolidate VirtualServices; every route rule is latency debt.
• Keep eBPF-based observability as ground truth for syscall-level latency, not only mesh metrics.

Four milliseconds p99 is not free. It is also not a reason to skip mTLS on everything — it is a reason to measure, exempt surgically, and stop pretending the sidecar is invisible.