ML Drift Monitoring in Production: What We Measure and Why
Feature distribution drift is not the same as label drift and neither is the same as business metric degradation. Confusing them leads to alert fatigue.
Ship log — ML drift monitoring, prod rollout week 8
Feature distribution drift, label drift, and business metric degradation are three different failure modes. We treated them as one alert for two years and burned out the on-call rotation. This document is what we measure now, on a fraud-scoring model (XGBoost 1.7, scikit-learn 1.3 pipeline) serving ~2M requests/day on AWS SageMaker Real-time (ml.c6i.xlarge), Python 3.10.
The confusion that caused alert fatigue
| Signal | What moved | What we thought it meant |
|---|---|---|
PSI on transaction_amount | Black Friday | Model broken — retrain |
| Label drift (chargeback rate) | New merchant cohort | Data pipeline bug |
| Approval rate drop | Policy team tightened rules | Model drift |
All three triggered the same PagerDuty policy. None of the three root causes were model weights.
What we measure now
Tier 1 — Business outcomes (daily, dashboard only unless catastrophic)
- Approval rate vs 28-day rolling baseline
- Fraud loss rate (lagged 14 days for chargebacks)
- Manual review queue depth
Alert: 3σ deviation sustained 2 days and no known policy change ticket linked.
These are not model metrics. They are the only metrics the CFO cares about.
Tier 2 — Label and outcome drift (weekly batch)
- Rolling chargeback rate by score decile
- Calibration plot shift (expected vs observed fraud rate per bin)
- Brier score week-over-week
Tool: internal Airflow 2.7 DAG, queries Snowflake, compares to MLflow 2.9 logged training baseline.
Alert: calibration error > 0.04 in any decile for 2 consecutive weeks → schedule retrain review, not auto-retrain.
Tier 3 — Feature drift (streaming + daily)
Per feature:
- PSI (Population Stability Index) vs training reference — alert PSI > 0.25
- KS statistic for continuous features
- Missing rate and sentinel values (we use -999 for unknown merchant category)
Infrastructure: Evidently AI 0.4.14 reports embedded in Spark 3.4 daily job; streaming approximations with Datadog custom metrics for top 12 features (cardinality controlled).
Critical: PSI on merchant_mcc moved every holiday season. We seasonally adjust baseline — October reference for Q4, not January training snapshot.
Tier 4 — Model internals (on demand)
- SHAP value distribution shift (TreeExplainer, sample 10k/day)
- Score distribution shift (not PSI on output — plain histogram KL divergence)
No pages. Review queue for ML engineer.
What we explicitly do not alert on
- Single-feature PSI 0.15–0.25 without Tier 1/2 movement
- Latency p99 (separate SLO)
- Training-serving skew detected in unit tests (block deploy, not page)
Architecture sketch
SageMaker endpoint → Kinesis Firehose → S3 (feature logs, no PII raw PAN)
↓
Glue crawler → Snowflake
↓
Airflow: Evidently + custom SQL → MLflow + PagerDuty routing
Feature logs store hashed entity IDs and binned amounts — not production raw features for regulated fields.
Cardinality and cost controls
Evidently reports on all 847 training features produced Snowflake bills we killed in week 3. Tier 3 now monitors top 40 by SHAP importance plus any feature tagged compliance_critical in the feature store (Feast 0.34). Remaining features get weekly batch only.
Datadog custom metrics for streaming PSI use log-based metrics with 1-hour rollups — 12 metrics, not 847. Budget alert at $400/month for the drift pipeline alone.
Retrain decision matrix
| Signal combo | Action |
|---|---|
| Tier 3 only, PSI < 0.35 | Log, review next weekly |
| Tier 3 + Tier 2 | ML on-call investigates within 24h |
| Tier 1 any | Incident channel, freeze auto-retrain |
| Tier 2 calibration only | Schedule retrain eval, no prod deploy |
Auto-retrain is disabled. Humans approve every model promotion after shadow comparison — we learned that the hard way in 2023 when auto-retrain amplified a labeling bug.
Incident example (2024-11)
PSI alert on device_fingerprint_entropy — 0.31. Tier 1 flat. Investigation: mobile app 4.2.0 changed fingerprint SDK, not fraud pattern. Retrain would have hurt — model learned old SDK quirks.
Action: feature flag to normalize SDK version in preprocessing; drift alert acknowledged with external_covariate_shift label in incident.io.
Runbook snippets
If Tier 1 + Tier 3 fire together: assume real drift or policy interaction. Roll back last model version via SageMaker alias (we keep live and previous aliases).
If Tier 3 only: check upstream schema PRs, holiday calendar, marketing campaigns.
If Tier 2 only: label delay — chargebacks lag; verify not partial label backfill.
Relation to other ML prod work
Quantization changed our score distribution without changing business metrics initially — see quantization war stories for why internal histograms lie after INT8.
Paper benchmarks rarely document seasonality — reproducibility checklist informs what we log at training time for drift baselines.
What I'd do next
Implement cause-specific alert routes in PagerDuty — drift/feature, drift/label, drift/business — different runbooks, different severity.
Experiment with conformal prediction bands on approval rate rather than PSI for executive-facing monitoring — one number, easier conversation.
Backtest: would this tier system have caught our 2023 false positive spike? Running that analysis on historical Snowflake exports now.
Manish Bookreader
Electronics enthusiast, Embedded Systems Expert, Linux/Networking programmer, and Software Engineer passionate about AI, electronics, books, and cooking.
