reranking

Two-stage retrieval

First-stage retrieval (keyword or vector search) is built for speed. It scans millions of passages and returns a rough shortlist, but it judges each document shallowly, either on keyword overlap or on a single compressed vector, so nuance gets lost. Reranking is the precision pass over that shortlist. A cross-encoder reads the query and one candidate together in the same input, runs a full inference step, and returns a relevance score. Because it looks at the pair jointly instead of comparing two pre-computed vectors, it catches meaning that first-stage retrieval misses.

The tradeoff is cost. A reranker runs one model pass per candidate, so it is far slower than a vector lookup. That is why systems retrieve many candidates cheaply (say the top 100), then rerank only that set down to the handful an LLM will actually read. Nogueira and Cho demonstrated the pattern in 2019, using BERT to rerank BM25 results and beating the previous state of the art on MS MARCO by 27% relative in MRR@10.

A typical call returns scored indexes:

rerank(query, documents) -> [{index: 3, score: 0.94}, {index: 7, score: 0.88}, ...]

Those scores decide which passages survive. Anything that does not make the reranked cut never reaches the answer, no matter how well it matched on keywords. In practice, reranking is where relevance actually gets judged, one query-document pair at a time.