A Score Ceiling in Reciprocal Rank Fusion Detects Zero Overlap, and Found a Dead Retriever

Benjamin Meadows · Independent Researcher

Abstract

A hybrid search system fuses a keyword retriever and a vector retriever with reciprocal rank fusion (RRF). In the system described here, the keyword retriever returned no results for most natural-language queries from the system's first version until 2026-08-18, and nobody noticed: the vector retriever always filled the result list, and the usual spot checks used short keyword queries, which still matched. The failure was visible the whole time in the fused scores. RRF credits a document 1/(k+r) from each retriever that ranks it at position r, so with per-retriever depth K ≤ k a document found by two retrievers scores above 1/k and a document found by one scores at most 1/k. A result list whose maximum fused score is 1/k therefore contains no document that both retrievers returned. That is a statement about overlap, not about either retriever: two working retrievers can return disjoint lists, and in the incident four of ten queries did. Across a set of queries, a ceiling that never lifts means no document is ever reinforced by two retrievers, and the per-retriever row counts, which the ceiling tells you to go and read, say why. I give the derivation and its limits, the bug (quoting every token of an SQLite FTS5 query, which FTS5 reads as an implicit AND), before-and-after aggregates from the production system, two negative results measured beside the dead leg, and a reproduction on two BEIR collections with two encoders: the builder returns nothing for 98.0% of SciFact's sentence queries and 50.8% of NFCorpus's title queries, the ceiling rate tracks the empty-leg rate to within one percentage point, and repairing the leg restores overlap, raises recall at depth for the weaker encoder, and does not improve nDCG@10 for either.

Code, results, per-query diagnostics and this paper's source: https://github.com/TheBenMeadows/dead-retriever

Cite

@misc{meadows_2026_23331139,
  author    = {Meadows, Benjamin},
  title     = {A Score Ceiling in Reciprocal Rank Fusion Detects Zero Overlap, and Found a Dead Retriever},
  month     = oct,
  year      = 2026,
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.23331139},
  url       = {https://doi.org/10.5281/zenodo.23331139},
}