ADR 0003 — Exact rescoring: fp16 re-rank for vectors, two-stage full-query BM25

Status: accepted · Date: 2026-08-23

Context

The 2.55M-row arXiv benchmark (bench/results-arxiv.md, real abstracts, 1024-d) exposed two recall gaps vs Weaviate that the 553k email benchmark (ADR 0001) never showed:

Metric	raggio	raggio-ivf	weaviate
Recall@10 vs exact	0.949	0.960	0.995
Hybrid text-hit@10	0.732	0.732	0.978

Root causes, both confirmed with read-only probes against the live bench data (DGX):

1. Vector recall loss is pure 4-bit quantization ranking error. An offline rebuild of the flat 4-bit index scored base recall@10 = 0.9650; re-ranking the top-C quantized candidates against the retained fp16 originals reached recall@10 = 1.0000 already at C=20 (stable through C=200) at ~0.1–0.3 ms/query. The fp16 blobs (ADR 0002, kept for index rebuilds) were never consulted at query time — the one step every comparable engine ships (Qdrant rescore+oversampling, Weaviate rescoreLimit, Milvus refine_k, ScaNN reordering, DiskANN).

2. Hybrid text-hit loss is the BM25 token pruning. The 2%-df budget (ADR 0001) was validated on the email corpus, whose text was constant boilerplate + path tokens; on real text it collapses: median arXiv title = 10 tokens, median kept after pruning = 3; 32% of titles keep ≤2 tokens (300-title sample). The BM25 leg then ranks the target by 1–3 rare tokens. Weaviate never drops query terms — BlockMax WAND skips documents via score upper bounds — and fuses 100-deep hybrid legs (hard-coded DefaultQueryHybridMaximumResults = 100).

FTS5 facts that constrain the fix (verified against fts5 source + measured on a 300k-row table):

• No WAND / dynamic pruning: ORDER BY rank LIMIT n scores every row matching the OR expression; bm25() additionally walks each phrase's full posting list once per query for IDF. A ranked full-OR title query on the 2.5M-row corpus takes seconds-to-minutes. Query structure is the only latency lever FTS5 offers.
• Never rank a rowid-restricted MATCH: bm25()'s per-query IDF cache is rebuilt on every xFilter call, so MATCH ? AND rowid IN (...) ORDER BY rank re-walks every phrase's posting list per candidate (measured 400×: 1155 ms vs 2.7 ms for 200 candidates). Rowid pushdown is fast (~13 µs/candidate) for membership only.
• Python-side BM25 over candidates is the sanctioned pattern (SQLite maintainers' guidance; APSW ships a reference port). FTS5's bm25 is reproducible exactly: k1=1.2, b=0.75 hardcoded, IDF = ln((N−df+0.5)/(df+0.5)) clamped to 1e-6.
• AND queries are intersection-bounded (doclist rowid seeks): cost tracks the rarest term even when co-terms are near-universal — cheap, high-precision candidate generation.

Decisions

Decision	Rationale / evidence
fp16 rescore of vector candidates — the quantized index over-fetches C = min(max(floor, 2n), 2000, N) (floor 50 at 4-bit, 200 at 2-bit), candidates re-ranked by exact cosine against the fp16 blobs; missing/corrupt blobs keep their quantized score	Probe: top-20 already contains the exact top-10 at 2.55M rows (recall@10 1.000). Cross-engine defaults bracket the multiplier (Weaviate rescoreLimit 20 for 8-bit RQ, Qdrant 3–4× oversampling for 1-bit BQ)
Rescore inside the existing batched-scan thread with one union blob fetch per batch (chunked ≤512 params)	Per-waiter round-trips would serialize sqlite point lookups behind the shared kernel pass and cut concurrent QPS
Rescore applies to all three search paths (batched scan, allowlist/filtered, sibling expansion)	One helper, consistent scores; sibling allowlists are doc-sized so cost is ~zero
Decoded fp16 candidates are renormalized before the dot product	fp16 round-trip drifts norms ~1e-3 — sub-point recall noise for near-ties
Two-stage BM25 when pruning dropped tokens: stage 1 = pruned-OR ranked LIMIT 500 ∪ AND-of-all-tokens unranked LIMIT 1000 (both under scope/filter); stage 2 = full-query BM25 in Python over the candidates (FTS5-parity k1/b/IDF, df from the shared df cache, avgdl from a churn-invalidated 256-doc sample)	End-to-end probe on the live 2.55M meta.db: target-doc hit@5 0.975 / hit@10 0.983 / hit@50 1.000 vs 0.732 today (Weaviate fused: 0.978). Unranked AND avoids the IDF-walk footgun; the AND cap is rowid-ordered (documented ceiling — stage 1a's rarest-token guarantee covers only-common-token queries)
Nothing-pruned queries (small corpora, trigram collections) keep the single FTS5 query	It already ranks the full query; zero added cost for the common small-collection case
SDM-lite proximity bonus in the Python scorer (ordered query bigrams, BM25-saturated, weight 0.2)	Metzler & Croft sequential-dependence: +5–9% MAP; recovers title-as-phrase known-item queries that pure BM25 legitimately ranks below term-stuffed docs. Caveat: canonical SDM uses a lower ordered-window ratio (~0.12) and true bigram statistics; 0.2 with mean-of-unigram-IDFs is unvalidated beyond this corpus and buys only ~+0.008 text-hit here
Hybrid leg depth 50 → 100	Weaviate parity (it fuses 100-deep even for k=10). Contributes ~zero to the text-hit metric (with disjoint legs, RRF k=60 decides the fused top-10 inside each leg's top-5); kept for general fused-tail quality at near-zero vector-leg cost
Pruning (ADR 0001) stays, demoted to candidate generation	Still the only bound on FTS5's rank-pass cost; it just no longer decides final ranking

Rejected

• WAND / BlockMax WAND in Python — FTS5 exposes no per-term cursor with skipping; a Python reimplementation over fts5vocab would be slower than FTS5's C scan.
• APSW custom rank function — swaps the sqlite driver to gain in-SQL ranking that the Python rescorer already provides.
• BM25F title column / separate title FTS index — schema change + 2.5M-row reindex; per-field weighting can be added to the Python rescorer later without reindexing.
• Relative-score fusion / weighted RRF — the fusion formula is not the gap (Weaviate's rankedFusion at alpha 0.5 is the same k=60 RRF); candidate future API, separate decision.
• SOAR spilled assignments, per-cell shard calibration, kNN-graph expansion — heavier IVF recall levers; the rescore already removes the quantization loss they mostly target.

Post-implementation audit (anti-benchmaxxing)

An independent ablation on the live bench db (120 real titles, target-in-text-leg-top-5, which is what the fused metric reduces to) decomposed the text-hit gain:

Variant	top-5 rate
pre-change (pruned-token FTS5 ranking)	0.732
two-stage full-query BM25 only	0.933
+ AND-of-all-tokens candidate branch	0.975
+ SDM bigram bonus (shipped)	0.983
Weaviate (published)	0.978

Honest readings that follow:

• Full-query rescoring is the fix (~80% of the gain) and generalizes; it repairs a self-inflicted defect, not a benchmark quirk.
• The AND branch is at its best case here: bench queries are exact titles and every doc's text starts with its title, so the AND match set has median size 1 (the target) and a single typo empties it. It remains a legitimate general candidate source (conjunctive retrieval), but expect its contribution to shrink on paraphrased/typo'd queries. A perturbed-query bench column is future work.
• The margin over Weaviate (0.984 vs 0.978) is 3 queries of 500 — within noise. Claim parity, not victory. The SDM bonus supplies that margin (+0.008) and its weight (0.2) exceeds the canonical SDM ordered-window ratio (~0.12); it awaits cross-corpus validation.
• Recall@10 = 1.000 means "quantization no longer costs recall on this corpus", not "beats HNSW". The over-fetch floors were validated on one corpus/model/dim; a corpus whose quantized top-2n misses the exact top-n would degrade silently — a per-request rescore-depth knob is the designated escape hatch if a real corpus ever needs it.
• Tuning provenance: the two-stage constants were selected on an independent seed-7 random-title sample (bench/text_probe.py), not on the published seed-42 bench queries; probe scripts are committed under bench/.
• The re-run's cold-start/memory numbers were reuse artifacts (warm page cache, compacted files); the published tables keep the original matched-conditions rows for metrics whose code did not change.

Consequences

• Vector scores are exact cosine similarities; text scores are positive BM25 points (Python provenance on the two-stage path). Ordering conventions unchanged.
• Vector queries pay one round of fp16 point lookups (~50–200 blobs); text queries that trigger stage 2 pay candidate text fetch + Python scoring (~tens of ms at 2.5M rows). Benchmarked before/after in bench/results-arxiv.md.
• The pre-existing hybrid p99 tail (a near-universal guarantee token forcing a large stage-1a rank pass) is unchanged — known future latency item, deliberately not coupled to this change.

Addendum 2026-09 — native stage 2

Stage 2 now runs in the raggio_native extension when it is installed (the image always builds it), with the Python scorer kept as its bit-identical reference and as the fallback (NATIVE_BM25=0): see ADR 0004. The stage-2 cost above describes the Python scorer, measured since on the DGX at 31.5 ms p50 per query against 1.5 ms native. Records, order and scores are unchanged by the scorer. The candidate prune now tokenizes like stage 2 (ADR 0004, spec D14), which changes the candidates of queries holding _ or compatibility characters (ligatures, superscripts, math alphanumerics). A query term in such a form no longer matches the same form in a document, because FTS5 indexes it as written (ADR 0004). The pre-existing hybrid p99 tail that ADR 0003's Consequences list as unchanged was the \w+ prune keeping df-0 underscore tokens, not a guarantee token; the prune tokenizer change (D14, ADR 0004) addresses it.