Fusing two rankers without calibrating their scores

Sooner or later every search stack ends up with two retrievers. One matches words, one matches meaning. Each is better than the other on a different third of the queries, and neither is better often enough to delete.

So you combine them. And immediately hit the problem that makes hybrid retrieval annoying: the two rankers do not speak the same numerical language.

Why you cannot simply add the scores

A BM25 score is unbounded. It grows with query length, with term rarity, and with how many query terms a document happens to contain. A score of is meaningful only relative to the other scores for that query. Ask a different question and the whole scale moves.

A cosine similarity lives in and, in practice, in a much narrower band than that — a dense retriever might return everything between and .

Add those two together and BM25 wins every time, not because it is more often right, but because its numbers are bigger. You have not built a hybrid; you have built BM25 with extra steps.

There are two honest ways out. Make the scores comparable, or stop using them.

Option one: normalize, then add

CombSUM with normalization. Map each ranker’s scores onto a common range and sum them:

Two normalizations do most of the work in practice.

Min–max squeezes each result list into :

Simple, and fragile in a specific way: it is defined by the two most extreme documents in the list. One runaway top hit compresses everything below it into a narrow band near zero, and the ranker effectively stops contributing.

Z-score standardizes by the distribution instead:

Less hostage to a single outlier, but it assumes the scores are roughly symmetric around the mean. Retrieval scores usually are not — they have a long right tail of a few very good matches, which is precisely the region you care about.

Both share a deeper problem: the normalization depends on which documents came back. Retrieve 100 candidates instead of 10 and every normalized score shifts, even though no document changed.

Option two: throw the scores away

Reciprocal rank fusion ignores the scores entirely and uses only position:

That is the whole algorithm. For each ranker that returned the document, add the reciprocal of its rank, offset by a constant .

It looks too crude to work, and it consistently beats carefully tuned score combination on heterogeneous rankers. The reason is that a rank is the one thing every ranker produces on the same scale. Rank 1 means the same thing coming out of BM25 as it does coming out of a neural model; a score of does not.

The constant controls how sharply the top of each list dominates. With the first result is worth twice the second and ten times the tenth — one ranker’s confident top hit can carry the fused list on its own. As grows the gaps flatten, and what matters becomes agreement: documents both rankers liked somewhat beat a document one ranker loved. The usual default is , which is very flat; smaller values are often better when one ranker is genuinely stronger.

Weighted RRF adds the obvious knob:

Now you can say “trust the lexical ranker 70% of the time” without having to know anything about its score distribution.

Try it

Below, a real BM25 implementation and a real concept-overlap retriever run over the same twelve-document corpus, in your browser. Type a query, pick a fusion method, and drag and the weight balance.

Things worth noticing

Try example 3. “Stop the feed showing the same thing over and over” contains not one word from the document about filter bubbles and diversity. BM25 scores it zero. The concept ranker finds it immediately. This is the entire argument for hybrid retrieval in one query.

Now drag from 1 to 60 on that query. At , whichever ranker put something first tends to keep it first. At the fused order is dominated by documents that both rankers ranked reasonably — consensus beats conviction.

Switch between min–max and z-score with a query that matches only one ranker. When a list has one or two hits, its normalization is degenerate — min–max maps the single result to 1.0 regardless of whether it was a good match. RRF has the same failure in a milder form, and neither method can rescue you from a ranker that should have abstained.

What I would actually reach for

Start with RRF. It has one parameter, it cannot be broken by a score distribution you did not anticipate, and it gives you a working hybrid in an afternoon. Tune on a held-out set — the default of 60 is a starting point, not a law.

Move to weighted score fusion when you have evidence that one ranker is systematically better and you are prepared to re-tune the normalization when either retriever changes. That second clause is where the cost lives. A model upgrade shifts the score distribution, the old normalization constants quietly stop fitting, and relevance degrades without anything appearing to break.

Rank-based fusion has no constants to go stale. For most teams that is worth more than the last two points of NDCG.