Quantization War Stories: INT8 and the Accuracy Cliff I Didn't Predict
Post-training quantization to INT8 worked beautifully on the benchmark — and lost 4 points on our production distribution. An anatomy of the failure.
Post-training INT8 quantization of our customer-support LLM (Llama 3.1 8B fine-tune, AWQ via AutoAWQ 0.2.4, serving with vLLM 0.4.2 on L4 GPU) looked perfect on MMLU-style benchmarks — and cost us 4.2 points on our internal resolution-rate metric on real tickets. This is the anatomy of that failure.
Setup
- Base: Llama 3.1 8B Instruct, LoRA fine-tune rank 64 on 180k support tickets (de-identified)
- Quantization: group size 128, zero point, INT8 weights + FP16 activations (W8A16)
- Baseline FP16: vLLM, max seq 4096, batch 32
- Eval sets:
- Public: 5-shot MMLU subset, GSM8K 50 problems
- Production mirror: 2,400 held-out tickets stratified by product line, language, escalation history
Hardware: NVIDIA L4 24 GB, driver 535.104.05, CUDA 12.2.
Benchmark vs production
| Metric | FP16 | INT8 W8A16 | Delta |
|---|---|---|---|
| MMLU (5-shot) | 61.2 | 60.8 | -0.4 |
| GSM8K | 42.0 | 41.0 | -1.0 |
| Resolution rate (prod eval) | 78.4% | 74.2% | -4.2 |
| p95 latency (ms) | 890 | 520 | -42% |
| VRAM (GB) | 17.1 | 9.8 | -43% |
We shipped INT8 based on MMLU + latency win. Rolled back in 36 hours after canary showed live resolution drop.
Serving stack interactions
vLLM 0.4.2 batching amplified quant errors: prefill on long system prompts plus concurrent requests changed KV cache layout vs single-request eval. Our prod-eval script used batch size 1; canary used batch 32. Always match batching in eval gates.
Tensor parallel was not in scope (single L4), but AWQ weight layout interacted with Marlin kernels in later vLLM versions — pin serving version in quant sign-off.
Calibration methodology notes
We now use 512–2048 sequences drawn from stratified production sample, min 50 per product line and language. Calibration runs on the same GPU arch as production — L4 vs A10G scale differences moved INT8 bounds enough to matter on one experiment (+0.6 resolution pts).
Outlier clipping during AWQ search: default settings clipped 0.1% of activations; our ticket bodies have heavy tails from pasted stack traces. Raising clip ratio to 0.5% helped without full FP16 fallback.
Why the cliff happened
Long-tail vocabulary in ticket bodies
Support tickets contain SKU fragments, log lines, base64-ish correlation IDs, and non-English mixed with English. Activation outliers in attention layers for rare token sequences quantize poorly with per-group scales.
Inspecting AWQ saliency maps: layers 18–22 (middle-high) had highest weight clipping rates on down_proj.
Calibration data mismatch
AutoAWQ calibration used 128 sequences from clean FAQ text. Production input distribution is nothing like FAQ text. Obvious in hindsight.
Re-calibration on 512 real ticket prefixes (truncated, PII scrubbed) recovered 2.1 points — still not enough.
Instruction-following fragility
Our system prompt is 1,800 tokens — tool definitions, policy citations, tone rules. Quantization error compounds across long prefills. FP16 tolerates small per-layer noise; argmax paths on policy-critical tokens flipped on ~3% of canary requests.
Example failure mode: model selects wrong refund policy section — grammatically fine, business wrong.
Metrics that hid the problem
Token-level perplexity on FAQ: INT8 within 2% of FP16. Useless for tool-use accuracy.
We now require task-specific eval passing gates, minimum sample 1,000, before any quant deploy.
What worked partially
- GPTQ with actorder on same calibration: 76.8% resolution — better, still below bar
- Mixed precision: layers 18–22 FP16, rest INT8 — 77.9%, 11 GB VRAM, acceptable for one region canary
- W4A16 (4-bit): do not ask — 68% resolution, instant rollback
Production gates today
- Prod-eval delta ≤ 0.5 points vs FP16
- No increase in escalation-to-human rate > 1% absolute
- Adversarial set: 200 tickets tagged "policy edge case" — zero tolerance for regression
- Shadow mode 72 hours minimum, compare decisions not just text similarity (BLEU lied too)
Tradeoffs I'd accept now
For draft replies (human approves before send): INT8 with recalibrated weights — latency matters, errors caught by human.
For autonomous refunds under $50: FP16 or mixed precision only. Cost of error >> GPU savings.
L4 at INT8 saves ~$4.2k/month per replica at our traffic. One PR crisis from wrong policy quote costs more.
Vendor and framework version matrix
| Component | Version pinned | Notes |
|---|---|---|
| AutoAWQ | 0.2.4 | newer 0.2.6 changed clip defaults |
| vLLM | 0.4.2 | Marlin kernels experimental |
| CUDA | 12.2 | driver 535+ required |
| Transformers | 4.38.2 | tokenizer merge rules affect rare tokens |
Pin everything in serving Dockerfile; diff container SBOM on every quant experiment.
Links forward
Drift monitoring caught the resolution drop before finance did — tier system in ML drift monitoring.
When evaluating published quant results, assume calibration split is wrong until proven — overlaps with paper reproducibility notes.
Next experiments
- FP8 (Hopper only) on H100 pool for training-serving parity test
- Knowledge distillation to smaller FP16 student instead of quantizing large model — runs Q2 on A10G budget
Edge deployment comparisons (Pi 5, Hailo) in edge inference benchmarking use different quant stacks — numbers do not transfer.
Manish Bookreader
Electronics enthusiast, Embedded Systems Expert, Linux/Networking programmer, and Software Engineer passionate about AI, electronics, books, and cooking.
