Fusing two rankers without calibrating their scores

Sooner or later every search stack ends up with two retrievers. One matches words, one matches meaning. Each is better than the other on a different third of the queries, and neither is better often enough to delete.

So you combine them. And immediately hit the problem that makes hybrid retrieval annoying: the two rankers do not speak the same numerical language.

The two retrievers, briefly

Skip ahead if these are familiar.

Lexical (BM25)

Matches the actual words. It counts how often each query term appears in a document, weights rare terms more heavily than common ones, and penalises long documents so they cannot win by sheer size. It is exact, fast, needs no training, and explains itself. It cannot match a word it has never seen: search for “laptop” and a page that only ever says “notebook” scores zero.

Dense (embeddings)

Encodes the query and every document into vectors, and matches by direction. It understands that laptop and notebook are the same thing, and that “stop the feed repeating itself” is about diversity. It is also confidently vague — it will happily return something in roughly the right area when the exact answer was right there, and it needs a model, an index and a re-encoding pipeline.

Their errors are close to uncorrelated, which is the entire reason to run both. Where one is confused, the other frequently is not.

Why you cannot simply add the scores

A BM25 score is unbounded. It grows with query length, with term rarity, and with how many query terms a document happens to contain. A score of is meaningful only relative to the other scores for that query. Ask a different question and the whole scale moves.

A cosine similarity lives in and, in practice, in a much narrower band than that — a dense retriever might return everything between and .

Add those two together and BM25 wins every time, not because it is more often right, but because its numbers are bigger. You have not built a hybrid; you have built BM25 with extra steps.

There are two honest ways out. Make the scores comparable, or stop using them.

Option one: normalize, then add

CombSUM with normalization. Map each ranker’s scores onto a common range and sum them:

Two normalizations do most of the work in practice.

Min–max squeezes each result list into :

Simple, and fragile in a specific way: it is defined by the two most extreme documents in the list. One runaway top hit compresses everything below it into a narrow band near zero, and the ranker effectively stops contributing.

Z-score standardizes by the distribution instead:

Less hostage to a single outlier, but it assumes the scores are roughly symmetric around the mean. Retrieval scores usually are not — they have a long right tail of a few very good matches, which is precisely the region you care about.

Both share a deeper problem: the normalization depends on which documents came back. Retrieve 100 candidates instead of 10 and every normalized score shifts, even though no document changed.

Option two: throw the scores away

Reciprocal rank fusion ignores the scores entirely and uses only position:

That is the whole algorithm. For each ranker that returned the document, add the reciprocal of its rank, offset by a constant .

It looks too crude to work, and it consistently beats carefully tuned score combination on heterogeneous rankers. The reason is that a rank is the one thing every ranker produces on the same scale. Rank 1 means the same thing coming out of BM25 as it does coming out of a neural model; a score of does not.

The constant controls how sharply the top of each list dominates. With the first result is worth twice the second and ten times the tenth — one ranker’s confident top hit can carry the fused list on its own. As grows the gaps flatten, and what matters becomes agreement: documents both rankers liked somewhat beat a document one ranker loved. The usual default is , which is very flat; smaller values are often better when one ranker is genuinely stronger.

Weighted RRF adds the obvious knob:

Now you can say “trust the lexical ranker 70% of the time” without having to know anything about its score distribution.

RRF on four documents, by hand

Say BM25 and the dense retriever return these two orderings, and take .

Document	BM25 rank	Dense rank			RRF	Fused
A	1	8	0.01639	0.01471	0.03110	1
B	4	2	0.01563	0.01613	0.03176	—
C	2	—	0.01613	0	0.01613	3
D	9	3	0.01449	0.01587	0.03036	4

Look at what happened. B — which neither ranker put first — comes out on top, because both rankers liked it. C was BM25’s second choice and drops to third, because the dense retriever did not return it at all. And A, a confident number one for BM25, only just holds on.

That is the behaviour rank fusion is for: it rewards agreement. Change to 1 and rerun the arithmetic and A wins comfortably ( from BM25 alone), because a small makes the top of each list enormously valuable. The constant is a dial between conviction and consensus, and it is the only thing there is to tune.

Run each retriever independently and take its top (typically 50–200; the depth matters, see below).
Resolve documents to a single stable id across both lists — the same document under two ids is fused with itself and wins on nothing.
For each document, sum over the retrievers that returned it. A retriever that did not return it contributes zero, not a penalty.
Sort by the sum, break ties on the strongest single retriever’s rank, and cut to the size the next stage wants.

Try it

Below, a real BM25 implementation and a real concept-overlap retriever run over the same twelve-document corpus, in your browser. Type a query, pick a fusion method, and drag and the weight balance.

Two rankers and their fusion, computed live. The left two columns are what each retriever returns on its own; the right column is the fused list. Documents that moved because of fusion are marked.

The second ranker here matches concepts through a hand-written alias table, not a neural embedding model — nothing on this page downloads a model. Its behavior is what matters for the demonstration: it catches vocabulary BM25 misses entirely and is hopeless at precision, which is exactly the trade a dense retriever makes. The fusion math is identical regardless of what produced the two lists.

Things worth noticing

Try example 3. “Stop the feed showing the same thing over and over” contains not one word from the document about filter bubbles and diversity. BM25 scores it zero. The concept ranker finds it immediately. This is the entire argument for hybrid retrieval in one query.

Now drag from 1 to 60 on that query. At , whichever ranker put something first tends to keep it first. At the fused order is dominated by documents that both rankers ranked reasonably — consensus beats conviction.

Switch between min–max and z-score with a query that matches only one ranker. When a list has one or two hits, its normalization is degenerate — min–max maps the single result to 1.0 regardless of whether it was a good match. RRF has the same failure in a milder form, and neither method can rescue you from a ranker that should have abstained.

Four things that decide whether this works in production

Retrieval depth is a real parameter. Each retriever’s top is the pool fusion gets to work with, and a document at rank 51 with contributes nothing at all — it is indistinguishable from a document the retriever hated. Going deeper costs latency and lets more noise in; going shallow throws away the complementarity you built the hybrid for. Tune on the same held-out set you tune on, and expect the answer to be larger than feels comfortable.

Deduplication happens before fusion, not after. If the same product appears under two SKUs, or the same document under two chunk ids, both copies collect their own reciprocal ranks and neither gets the other’s. The fused list then puts the duplicate pair below a single document both rankers agreed on, which is the opposite of what you wanted. Resolve to a canonical id first.

Missing is not the same as bad. A retriever that did not return a document contributes zero to its RRF sum. That is the correct default — abstention is not a negative vote — but it does mean a document only one retriever knows about needs to be very highly ranked by that one to compete. If one of your retrievers has poor coverage of a whole content type, fusion will quietly suppress that type and no metric will name the cause.

Fusion is not a reranker. RRF has no features, no training and no way to learn that this query is the kind where the dense retriever should be trusted. Once you have enough labelled data, a learned reranker over the fused candidate set will beat any fusion rule — and it will beat it while still using fusion for candidate generation. These are different stages, not competing choices.

What I would actually reach for

Start with RRF. It has one parameter, it cannot be broken by a score distribution you did not anticipate, and it gives you a working hybrid in an afternoon. Tune on a held-out set — the default of 60 is a starting point, not a law.

Move to weighted score fusion when you have evidence that one ranker is systematically better and you are prepared to re-tune the normalization when either retriever changes. That second clause is where the cost lives. A model upgrade shifts the score distribution, the old normalization constants quietly stop fitting, and relevance degrades without anything appearing to break.

Rank-based fusion has no constants to go stale. For most teams that is worth more than the last two points of NDCG.