Paper Reproducibility in ML: A Checklist From 30 Replication Attempts

Over 30 months I attempted to reproduce 30 published ML results — papers from NeurIPS, ICML, ACL, and internal company reports. 11 fully replicated within 2% of claimed metrics, 9 partially replicated, 10 failed. This checklist is what I wish each paper had included, and what we now require before citing a result in a production decision.

Replication scorecard (n=30)

Outcome	Count	Typical blocker
Full replicate	11	Had code + seeds + data spec
Partial	9	Missing one of: data filter, aug policy, eval script
Failed	10	Hardware-dependent claim or hidden preprocessing

Failure is not always fraud — often underspecified pipeline.

Checklist — before trusting a number

Data (most failures here)

• Exact dataset version — not "ImageNet" but ILSVRC2012, checksum, train/val split file hash
• Filtering rules — "removed corrupted images" hides 400k drops
• Train/val/test leakage checks — duplicate near-duplicates across splits?
• Label definition — especially NLP: who annotated, inter-rater κ
• Preprocessing code — not "standard tokenizer"; exact vocab.json revision

Paper #17 claimed 94.1 F1; we hit 89.3 until we found they lowercased before tokenization but reported BERT uncased — docstring in repo, not paper.

Hyperparameters

• Full table: lr, schedule, warmup steps, batch size per GPU, grad accum, weight decay
• Seed list — mean ± std over ≥3 seeds, not cherry-picked best
• Early stopping criterion — val loss vs downstream metric
• Optimizer ε, β values — AdamW defaults changed between PyTorch 1.13 and 2.1

We log pip freeze and torch.__version__ in MLflow now — non-negotiable.

Compute and implementation

• GPU model, driver, CUDA, cuDNN, framework versions
• Effective batch size and whether BatchNorm stats synced across GPUs
• Mixed precision: FP16, BF16, loss scaling?
• Custom CUDA kernels — flag if results depend on them (flash-attn version!)

Three "failed" replications were BF16 vs FP16 numerical differences on attention softmax — within 1.5 points, paper did not specify.

Evaluation

• Eval script committed, not notebook cell 47
• Test set frozen before hparam search — verify commit timestamps
• Metric implementation — sklearn f1_score(average=?) matters
• Statistical test — confidence intervals, not point estimates

Release artifacts

• Weights hosted with SHA256
• Config YAML that loads in one command
• License on data and weights — blocked two enterprise replications

Process we use internally

1. 48-hour spike — one engineer, one GPU, reproduce val metric only
2. Replication doc — PASS / PARTIAL / FAIL with diff table
3. Production citation gate — FAIL blocks architecture review unless labeled "directional only"

Tools: MLflow 2.9, DVC 3.38 for data hashes, reproducibility.yml in repo root (GitHub Actions runs pytest tests/repro/ on schedule).

Patterns in the 10 failures

1. Unreported data cleaning (4 papers) — outlier removal on labels
2. Best-of-5 seed reporting (2 papers) — we reproduced best seed, not mean
3. Eval on train distribution (2 papers) — test set from same crawl as train
4. Hardware-specific kernel (2 papers) — TPU numbers not reproducible on our A100s without 3 weeks porting

What authors can do (we ask reviewers for this)

• Appendix: one-page "Reproducibility Card" with command:

python train.py --config configs/paper.yaml  # expect val F1 0.941 ± 0.008

• Release container digest: ghcr.io/lab/paper:v1@sha256:...
• Negative results on seeds — builds trust

Connection to production ML

Quantization papers rarely report calibration data distribution — we learned that painfully in quantization war stories.

Drift baselines assume training snapshot is documented — drift monitoring requires the same metadata discipline.

What I'd do next

Submit a NeurIPS reproducibility challenge entry template to our research partnerships — standardized card, bot checks for missing seeds.

Automate first-pass replication from Hugging Face model cards: parse training hyperparameters, compare to paper table, flag deltas >5%.

Stop citing leaderboard numbers in RFCs without linking internal replication status — same bar as writing RFCs that get read evidence requirements.

Time accounting (honest)

Median time per replication attempt: 16 engineer-hours when code exists, 40+ when only PDF. We budget one replication spike per quarter per team for papers that influence roadmap — not every arXiv preprint.

Partial replications often fail on random seed alone. Paper reported 92.3 ± 0.4 with n=3 seeds; we saw 91.1, 92.8, 90.6, 93.0, 91.9 — their std was understated, not our failure.

Internal policy (2025)

Papers cited in architecture reviews must include replication status badge: REPLICATED / PARTIAL / NOT ATTEMPTED / FAILED. Product managers cannot bypass — same gate as security review for external SaaS integrations.

We maintain a Notion table of 30 attempts; link from MLflow experiment notes when a paper inspired a hyperparameter choice.

Reviewers at external conferences: ask for artifact link in rebuttal — we have stopped accepting "will release upon acceptance" without timestamped tarball in supplemental.

Failed replications get tagged directional-only in our internal wiki — still citable for intuition, not for SLO numbers.

What authors can do (we ask reviewers for this)