Kafka Partition Strategy: When Consumer Lag Becomes Your Alert, Not Your Dashboard

Consumer lag is a lagging indicator. By the time it turns red on the Grafana panel, the ordering guarantees you thought you had may already be violated, and your dead-letter queue may be holding messages you cannot replay without corrupting downstream state.

We learned this on Kafka 3.6 (KRaft mode, no ZooKeeper) when a payments-adjacent topic went from green to 45 minutes of lag over a weekend — not because consumers died, but because we mis-partitioned for both throughput and ordering semantics.

The incident setup

Topic: order-events-v2, 24 partitions, replication factor 3, min.insync.replicas=2. Producers keyed by merchant_id to preserve per-merchant ordering. Three consumer groups:

• Fraud scoring (12 consumers, CPU-bound ML inference)
• Ledger projection (6 consumers, strict ordering per merchant)
• Analytics (Flink 1.18 job, event-time windows)

We sized partitions using a back-of-envelope: peak 8k msgs/sec, target 500 msgs/sec/partition → 16 partitions, rounded up to 24 for headroom. Standard stuff from the Confluent docs.

What we missed: fraud scoring and ledger projection both needed per-merchant ordering, but fraud was slower. When a hot merchant spiked, their partition lagged in the fraud group while ledger kept up — until ledger's state store compaction fell behind and we got cross-partition read skew in the admin UI.

Partition strategy decisions that actually matter

Partition key vs consumer count

Rule we now enforce: consumers in a group ≤ partitions, and if ordering matters, one slow consumer blocks the entire partition. We moved fraud scoring to a two-stage pipeline:

1. Fast path consumer: validate, enrich, write to fraud-candidates keyed by merchant_id
2. Slow path: separate topic, 48 partitions, keyed by hash of (merchant_id, event_id) to strip ordering requirement for ML

Ledger stayed on merchant_id key with 24 partitions and exactly 6 consumers — never scale consumers without re-partitioning.

When to alert on lag vs offset rate

Dashboard lag (records-end-offset - current-offset) lied to us during a broker rolling restart. Metrics dipped, came back, nobody paged.

We now alert on:

deriv(consumer_group_lag[5m]) > threshold
AND
consumer_group_lag > N minutes of peak throughput

For ledger: if lag > 90 seconds at p99 produce rate, page. For analytics: lag > 15 minutes, ticket (SLA is hours).

Also: lag per partition, not just aggregate. Aggregate lag hid a single partition stuck at 2M messages because one consumer repeatedly crashed on a poison pill.

Re-partitioning cost

We increased order-events-v2 from 24 → 48 partitions using Kafka's partition increase (keys still hash to new partition count — ordering breaks for in-flight keys during transition). We did it during a maintenance window with producers paused for 12 minutes. Downtime we could have avoided with better initial sizing.

Document in the runbook: increasing partitions does not split existing data; it only affects new messages.

Instrumentation we added

• Burrow (LinkedIn, v1.6) for consumer lag monitoring with partition-level granularity — Prometheus exporter sidecar
• OpenTelemetry spans on consume loop: process.duration histogram tagged by partition, not just topic
• Synthetic canary producer every 60s with {canary: true} — if canary lag > 30s, page before business traffic backs up

See eBPF observability in production for how we correlated kernel-level TCP retransmits with consumer poll timeouts — unrelated root cause, same weekend.

Ordering guarantees we stopped pretending to have

We wrote down explicit semantics per topic:

Topic	Ordering guarantee	Delivery
order-events-v2	Per merchant_id	At-least-once
fraud-candidates	None	At-least-once
ledger-commands	Per merchant_id	Effectively exactly-once via idempotent upserts

Anything not in the table defaults to no ordering. Engineers stopped assuming.

Producer-side settings we tightened

After the incident we audited producer configs cluster-wide:

• acks=all on financial topics — no acks=1 shortcuts left from early POCs
• enable.idempotence=true — eliminates duplicate sequence numbers from retries; required bumping max.in.flight.requests.per.connection discipline
• linger.ms=5 and batch.size=32KB — reduced partition hot-spotting from per-message flushes during merchant spikes
• Compression: lz4 — CPU cheaper than snappy on our ARM brokers; 22% byte reduction on JSON-heavy payloads

Hot-key mitigation beyond partitioning: jittered salting for analytics-only copies of merchant events (separate topic, key = hash(merchant_id + hour_bucket)). Ledger path never got salted — ordering non-negotiable there.

Replay discipline

When lag exceeded 30 minutes on ledger, we considered replay from earliest offset. Rejected: idempotent upserts handle duplicates but downstream notifications do not. Replay plan now requires:

1. Pause outbound webhooks at consumer
2. Replay with new consumer group ID
3. Diff projection against live table before cutover

Documented in Confluence — not in Kafka broker config, but lag response is incomplete without it.

What I'd do next

1. Partition sizing worksheet in every RFC: peak msg/sec, p99 process time, ordering scope, replay tolerance.
2. DLQ with quarantine: poison messages go to topic.DLQ with original partition logged; auto-skip requires two-person approval.
3. Lag SLO per consumer group, not per cluster — tied to incident response runbooks with pre-written "pause producers" steps.

Consumer lag on a dashboard is a autopsy metric. Alert on its derivative and on per-partition skew, or you are reading yesterday's problem.