Rerankers

A recruiter with 500 resumes does not read all 500 closely. They skim fast, keyword by keyword, down to a shortlist of 20, then actually read those 20 to decide the order. The first pass is quick and rough on purpose; the careful reading only happens on the few that survived it. Search does the same thing, and the careful second pass is called reranking.

Every earlier part of this series was about the first pass: BM25, vector search, hybrid. They are built to be fast and to not miss, pulling a few hundred maybe-relevant documents out of millions. What they are not built for is precise ordering. The top of that list is often in the wrong order, with a keyword-stuffed but useless document above the one you actually wanted.

Two encoders, two jobs

The reason first-stage retrieval ranks crudely comes down to how it reads.

Retrieval uses a bi-encoder. The query and each document are turned into vectors separately, and the score is how close those two vectors are. Because the document vectors can be computed ahead of time and stored, this is enormously fast: at query time you embed just the query and compare. The catch is that the model never looks at the query and the document together, so it misses fine-grained relevance.

Reranking uses a cross-encoder. The query and one document are fed into the model as a single joined input, so the model can attend across both at once and judge, in detail, how well this document answers this exact query. That is far more accurate. The catch is that nothing can be precomputed: you must run the model once per query-document pair, at query time.

That cost is the whole reason reranking is a second stage. Running a cross-encoder against all ten million documents per query would be as hopeless as brute-force exact search. Running it against the top 100 the first stage already found is cheap, and it buys most of the accuracy.

The two-stage pipeline

So production search and RAG almost always look like this: retrieve a few hundred candidates fast with BM25, vectors, or hybrid, tuned for recall so the right answer is somewhere in the set; then rerank the top of that set with a cross-encoder, tuned for precision so the right answer rises to the top; then return the top handful.

Watch it happen. The first-stage list is ordered by a cheap score; press Rerank and a cross-encoder rereads each result against the query and reorders:

Notice what moves. A document with almost no keyword overlap that first-stage buried can leap to the top once a model actually reads it against the query, and a keyword-heavy but off-topic result sinks. That correction is what a reranker is for.

In production

Reranking is a standard layer. Hosted rerank APIs like Cohere Rerank, and open cross-encoder models such as the ms-marco cross-encoders, drop in after first-stage retrieval, and in RAG a reranker before the top chunks reach the model is one of the highest-leverage quality upgrades available, because the model only ever sees the few passages the reranker floated to the top.

That closes the loop this series opened. Search is four steps, index, parse, match, rank, and a hundred choices about how to do each. Keyword, fuzzy, vector, approximate, hybrid, and reranking are all just different answers to "what counts as a match, and in what order." Underneath every one of them, it is still a catalog built so you never have to read the whole library.