How deep should the expensive stage look?

Every ranking stack that survives contact with production has the same shape: a cheap stage that looks at everything, and an expensive stage that looks at a shortlist. The question nobody quite answers is how long the shortlist should be.

The usual reasoning is that more is better and the limit is latency, so the answer is however many you can afford. That reasoning is wrong in an interesting way.

The setup

Stage one is a retriever — BM25, a two-tower model, or a fusion of several. It is fast, it sees the whole catalogue, and it is noisy.

Stage two is a reranker — a cross-encoder that reads the query and the document together, or a gradient-boosted model with expensive features. It is much more accurate and much more expensive, so it only ever sees the first results from stage one.

dominates. A batched cross-encoder is somewhere around a third of a millisecond per document on a GPU, so is about 105 ms of pure model time before anything else in the request has happened.

The curve

Below, 120 queries with 600 candidates each. Both stages are simulated, and every depth from 10 to 500 is evaluated exactly — the widget is not interpolating between a few samples.

Quality against reranking depth. The highlighted curve is the reranker you have selected; the faint ones are the other two. The dashed marker sits at the depth that maximises nDCG@10 for the current pair of stages, and the dot is where your slider is.

The default is a reasonable-looking production choice: an adequate reranker, a typical retriever, 300 candidates. It scores 0.850 on nDCG@10, costs 117 ms, and runs about $58 per million queries.

The best depth for that pair of stages is 103 — which scores 0.856, costs 48 ms, and runs $20 per million queries.

Better quality, 2.4× less latency, a third of the cost. Reranking 300 candidates is not a conservative choice that trades money for safety. It is worse on every axis at once.

Why more candidates can be worse

Switch the reranker to perfect and the curve becomes monotone: more depth is always better, forever, exactly as intuition says. So the decline is not a property of cascades. It is a property of imperfect rerankers, and it has a clean explanation.

Think about what happens when you extend the shortlist from 300 to 301. The new document is, by construction, one that stage one ranked 301st — so it is probably not very good. Two things can happen:

The upside

Occasionally the retriever was wrong and this document is genuinely excellent. The reranker recognises it and promotes it. You win.

The downside

Much more often the document is mediocre, and the reranker’s own error is large enough to score it above something in your top ten. You lose.

Both effects scale with depth, and they scale differently. The chance of finding a missed gem falls off fast — the retriever’s ranking is informative, so good documents are concentrated near the front. The chance of a noisy promotion does not fall off at all; every candidate gets an independent draw from the reranker’s error distribution. Past some depth the second term wins.

The location of the peak is a measurement. A curve that peaks early and falls steeply is telling you the reranker is not much better than the retriever. A curve that keeps rising to 500 is telling you the reranker is strong and the retriever is the bottleneck. Before tuning the depth, read which of those you have — the two situations call for opposite investments.

The first stage is the cheaper lever

Drag how noisy the cheap stage is down to 0.30 and watch the panels.

The optimum depth for the adequate reranker drops from 103 to 52, and the peak quality rises from 0.856 to 0.889. A sharper retriever made the system better and halved the serving cost, because the good documents are now near the front and there is nothing worth paying to look at further down.

This is the part that tends to get missed in latency reviews. The conversation is usually “can we afford 300?”, when the available move is “make stage one better and afford 50”. Improvements to the first stage compound: better quality at lower cost, in the same change.

The true top-10 handed over readout is the other half of that argument. At the default settings, 300 candidates contain only part of the true top ten — and the reranker cannot return a document it was never given. That number is a hard ceiling on everything downstream, and it belongs on the same dashboard as the final metric.

Getting the measurement right

Sweep the whole range once, offline. The curve is cheap: score a sample of queries at every depth using logged candidates. It is a batch job, not an experiment, and it gives you the peak and the shape in an afternoon.

Price the axis in the units you are actually constrained by. Latency and cost are different constraints and they bind at different times. A consumer search box is constrained by p99 latency; a nightly batch of recommendations is constrained by GPU-hours and does not care about latency at all. The same curve, two very different answers.

Remember the tail. The widget reports mean latency. Reranking documents has a distribution, and p99 is what pages people. If your infrastructure batches requests, a large also reduces how many requests fit in a batch, which degrades throughput faster than the linear model suggests.

Vary the depth per request if you can. Not all queries deserve the same budget. A head query with an obvious answer needs almost no reranking; an ambiguous tail query is where depth pays. Predicting which is which is its own model, but even a crude rule — shorter queries get less depth, queries with a confident stage-one top result get less depth — recovers most of the available saving.

Re-measure after every model change on either side. The optimum is a property of the pair. Ship a better retriever and the right depth falls. Ship a better reranker and it rises. Neither change tells you about the other, and a depth constant that was tuned two model versions ago is now just a number somebody typed.