Event Sourcing in Production: Three Years of Append-Only Truth

Three years into event sourcing for our order-fulfillment domain, I would still choose it for auditability—but not for every bounded context. Append-only truth made regulatory replay straightforward; schema evolution, projection rebuild times, and "where is the current state?" onboarding tax remain real. This is the honest ledger.

Why we chose it

Requirements: immutable audit trail (who changed shipping address when), temporal queries ("what did inventory look like Tuesday 14:00 UTC?"), and decoupled read models for warehouse tablets vs finance exports.

Stack: PostgreSQL 15 event store table, Kafka 3.5 for cross-service fan-out, Kotlin services, Axon-inspired patterns without Axon framework lock-in.

What worked

Audit and dispute resolution. Support pulls event ID evt_8f2a..., reconstructs state—no guessing from UPDATE logs that overwrote data.

Multiple read models. OrderSummaryProjection for API, PickListProjection for warehouse, RevenueRecognitionProjection for finance—all subscribed to OrderEvents topic with independent consumer groups.

Debugging production weirdness. Replay a single aggregate's events in staging reproduced a race we never saw in unit tests.

Kafka partition strategy mattered—see Kafka partition strategy and consumer lag. We partition by orderId hash; hot SKUs during flash sales still lagged pick-list projectors until we scaled consumers.

What hurt

Schema evolution

Events are forever. Renaming shippingLine1 → addressLine1 sounds trivial; every consumer deserializer must tolerate both for years. We adopted:

• Upcasters in projection code (version field per event type)
• Protobuf with reserved fields—better than JSON, still discipline-heavy
• Ban on "fix up" migrations that DELETE events—only compensating events

New engineers underestimate this. Onboarding doc now includes "adding an event field" checklist with 7 steps.

Replay performance

Full rebuild of PickListProjection from 890M events took 6.5 hours on a db.r6g.2xlarge read replica dedicated to replays. Production traffic could not share that instance—replay I/O saturated Postgres index bloat we had ignored on the events table.

Mitigation: snapshot + incremental replay from snapshot timestamp. Snapshots every 10k events per aggregate cut rebuild to 22 minutes.

Operational complexity

"Just query the database" became "which database?" Event store is source of truth; API reads projection; finance reads different projection that might be 30 s behind. Incident during lag spike: customer charged for cancelled order because payment projection hadn't caught OrderCancelled.

Fix: user-facing reads for money-touching flows go through synchronous command path that waits for critical projection catch-up or reads from strongly consistent store—not eventual warehouse list.

Event size creep

Started averaging 400 B JSON per event; three years later average 2.1 KB—embedded PDF metadata someone added "just once." Compacted topic helped Kafka retention cost; PostgreSQL store grew regardless.

Numbers (approximate, FY2025)

• Events stored: ~1.2B rows, 2.8 TB with indexes
• P99 command append latency: 18 ms
• P99 projection lag under normal load: 400 ms
• P99 projection lag during Black Friday: 45 s (bad)
• Engineers who would pick ES again for this domain: 6/8

CQRS coupling

Event sourcing implied CQRS; we did not need full CQRS everywhere. Inventory still uses traditional CRUD with outbox pattern—trying to force ES created fake "InventoryAdjusted" events nobody replayed.

What I'd do next

1. Event catalog as product—schema registry (Confluent 7.5) with compatibility CI on every PR.
2. Tiered storage—events older than 18 months to S3 Parquet; hot store keeps 90 days for replay SLA.
3. Kill partial ES in two remaining microservices that only use events as log fluff—migrate to outbox + CDC.

Team skills and hiring impact

Event-sourced systems reward engineers comfortable with temporal reasoning and idempotent consumers. We lost two senior hires during onboarding when they discovered "the database" was not one schema but twelve projections with different lag SLAs. Interview loop now includes take-home: implement compensating event for PaymentCaptured after OrderCancelled—filters candidates early.

Cost accounting

Storage grew faster than transaction volume—2.8 TB for 1.2B events because we never archived ProductCatalogUpdated high-frequency noise. FinOps dashboard now tracks $/million events by type; teams pay internal chargeback for events they emit (controversial, effective).

Event versioning workshop

Quarterly workshop: product + eng review new event types before merge. Rejected UserClickedButton event—analytics belongs in clickstream pipeline, not ES store. Gate reduced event volume 12% year two.

Read model rebuild runbook

Runbook steps: snapshot restore, replay from offset, validate row counts against checksum table, flip read alias—tested quarterly in staging. First production rebuild without runbook took 9 hours extra arguing about order of operations.

On-call and projection lag alerts

Pager fires when pick_list_projection_lag_seconds p99 > 30 for 5 min—correlates with customer-visible stale inventory before support queue spikes. Event sourcing operational cost is metrics on projections, not just append latency.

Event sourcing is a scalpel: excellent for audit-heavy, temporal domains; expensive as default CRUD replacement. Three years in, our best ROI came from narrowing scope, not expanding it.