Embedding Model Selection for a Production Search System

text-embedding-ada-002 is the default everyone reaches for. On our domain-specific retrieval task — internal engineering runbooks, 48k documents, technical vocabulary heavy on part numbers and acronyms — it ranked fourth out of five in our benchmark. A 384-dim open-weight model beat the 1536-dim OpenAI baseline on recall@10.

This is a lab note from Q1 2024 benchmarking, not a universal ranking.

Task definition

Corpus: 48,312 Markdown/HTML runbooks, postmortems, ADRs. Average doc length 890 tokens. Duplicates deduplicated by SHA-256.

Queries: 520 labeled pairs from support escalations (question → relevant doc IDs). Held-out 20% for eval. Labels by two senior engineers, Cohen's kappa 0.81.

Metrics: Recall@5, Recall@10, MRR@10. Also measured p95 embedding latency and $/1M tokens at our volume.

Vector store: pgvector 0.6 on PostgreSQL 16, HNSW index (m=16, ef_construction=64). Same index params for all models — fair comparison on search, not index tuning per model.

Models tested

Model	Dims	Open	Recall@10	p95 embed (ms)	Cost model
text-embedding-ada-002	1536	No	0.742	45 (API)	$0.10/1M tok
text-embedding-3-small	1536	No	0.761	38 (API)	$0.02/1M tok
bge-large-en-v1.5	1024	Yes	0.778	12 (A10G)	GPU amortized
e5-large-v2	1024	Yes	0.769	11 (A10G)	GPU amortized
all-MiniLM-L6-v2	384	Yes	0.791	3 (CPU)	Cheapest
gte-large-en-v1.5	1024	Yes	0.756	13 (A10G)	GPU amortized

Surprise: all-MiniLM-L6-v2 (384 dims) won on recall@10. Investigation: our queries are short (avg 12 tokens), often exact acronym matches (NVIC, EBPF, part numbers). Smaller model not better semantically — better on lexical overlap in embedding space for our query length distribution. Longer natural-language queries (subset n=80) favored bge-large.

Domain-specific findings

Acronyms and part numbers

OpenAI embeddings treat STM32F407 and STM32F405 as very similar (cosine ~0.94). Support queries often need the exact variant. Mitigation that helped all models:

• Query expansion via glossary lookup before embed (not model swap)
• Store product_sku as metadata filter, not embedded text

Model swap alone did not fix variant confusion.

Document length and chunking

Runbooks with code blocks embedded poorly when chunked at fixed 512 tokens — function signatures split across chunks. Structure-aware chunking (split on ## headers) improved recall@10 by 0.06–0.09 across all models. Chunking mattered more than model choice for procedural docs.

Fine-tuning vs off-the-shelf

We tested embedding fine-tune on 2,400 query-doc pairs (see RAG vs fine-tuning framework). bge-large fine-tuned (+0.04 recall@10) beat MiniLM off-the-shelf but lost to MiniLM + glossary expansion (+0.05). Fine-tune not worth ops burden at our scale.

Production decision

Shipped: all-MiniLM-L6-v2 on CPU (ONNX Runtime, 4 replicas) for default search. Fallback tier: bge-large for "research" search tab with longer queries. OpenAI text-embedding-3-small for one product area that wanted managed SLA and sends customer-facing text (different compliance boundary).

Index size: 384 dims → 60% smaller HNSW than 1536 dims. Insert throughput 2.3x faster on reindex.

Reindex operational cost

Embedding model swap is not flip-a-flag. Our reindex job:

• Duration: 48k docs × 890 tok avg, MiniLM on 4 CPU replicas → ~6 hours full rebuild
• Dual-write period: 72 hours with old+new indexes queried in shadow; compare recall on sampled queries
• Rollback: keep old HNSW index on separate tablespace until validation passes

Underestimating reindex time caused a weekend where search quality was inconsistent — new docs embedded with new model, old docs stale. Batch cron now tags docs with embed_model_version metadata; search filters mismatched versions during transition.

Query-side lessons

Short acronym queries (NVIC, I2C) benefited from hybrid BM25 more than embedding swap. We added ts_rank on title + glossary expansion before embed — recall@10 +0.04 on acronym subset without changing model. Embedding benchmark alone would have missed this.

Storage and index tuning notes

HNSW parameters (m=16, ef_construction=64) were not optimized per model — fair benchmark constraint. Production tuned ef_search=100 for MiniLM vs ef_search=128 for ada-002 dimensional density — latency/recall trade documented per index.

pgvector 0.6 IVFFlat tested on ada-002 index — build time 4x faster than HNSW but recall@10 dropped 0.09 on our eval. HNSW worth the build pain for offline batch reindex window.

Drift monitoring

Embedding model changes require reindex — not hot-swappable. We track ML drift on query distribution: if avg query length or acronym rate shifts, re-run offline eval before model swap.

What I'd do next

1. Hybrid search — BM25 (pg_trgm + tsvector) + vector, RRF fusion. Benchmark suggests +0.03 recall@10 for part-number queries regardless of embed model.
2. Re-evaluate text-embedding-3-large on long-query subset only — probably not full corpus.
3. Test multilingual-e5 for our growing APAC runbook translations.

Default embeddings are a convenience, not a recommendation. Benchmark on your query length, vocabulary, and chunking — the winner might be the smallest model in the list.