Offline said it would work

NDCG@10 up 1.2%. Two weeks of work, a clean offline evaluation, an approving review. It ships to 50% of traffic and the online metric does not move.

Everyone has a theory. The experiment was underpowered. It needs longer for novelty to wear off. The offline set is not representative. Some of those are sometimes true, and none of them is the structural answer.

They are not the same quantity

The structural answer is that the offline number and the online number are measuring different things, and the relationship between them is a correlation rather than an identity. Three separate mechanisms drive them apart.

• The proxy is not the goal NDCG against human judgments measures whether an assessor thinks the results answer the query. The online metric measures whether a real person, with their own intent and their own hurry, did something. These agree often and not always, and how often is a property of your product that you can measure but cannot assume.
• The evaluation data came from the system you are replacing Judged pools are assembled from what the current ranker retrieved. Click logs record what the current ranker showed. A candidate that agrees with the incumbent therefore scores well for reasons unrelated to quality — and one that finds something genuinely new gets no credit at all, because nobody judged it and nobody was ever given the chance to click it. This is position bias and selection bias arriving in your evaluation rather than your training data.
• The measurement is noisy Per-query NDCG varies enormously. On a few hundred judged queries the standard error is a large fraction of the effect you are trying to detect, so some of what you see is simply which queries you happened to judge.

So treat it as a screening test

A screening test does not tell you whether a patient has the disease. It tells you whether to run the expensive test. It is judged on precision and recall, and those are measurable.

The same framing works exactly here. An offline evaluation is a cheap test whose job is to decide which candidates deserve an online experiment. The useful questions are: of what clears the bar, how much is real? And of what is real, how much do we catch?

Below, 400 candidate ranking changes. Each has a true online effect that nobody in the simulation can see, and an offline measurement that is a distorted, noisy view of it. The threshold decides what ships.

What the defaults say

The defaults are deliberately ordinary: an offline metric that captures three quarters of the true effect, a moderate reward for resembling the incumbent, and 400 judged queries. The offline–online correlation comes out at 0.64 — which is a good offline setup, better than many teams have and roughly in line with what the search literature reports.

At that correlation, of the 124 candidates that clear the bar:

• 50% are genuine wins.
• 28% actively make the product worse.
• And 61% of the real wins made it through — the rest were rejected offline and never tested.

That is what a correlation of 0.64 buys. Not a prediction. A coin flip with a useful thumb on the scale.

The three levers, in order of payoff

Remove the confound before you buy more judgments. Drag reward for resembling the old system to zero and the correlation goes from 0.64 to 0.74, precision from 50% to 62%, and harmful ships from 28% to 16% — without a single extra judged query. In practice that means pooling candidates from every system under comparison before sending anything for judgment, and keeping a slice of randomised traffic so the log contains documents the incumbent would never have shown. It is the cheapest of the three and almost always the one skipped.

Then buy judgments, knowing the exchange rate. The noise falls as , so going from 100 to 400 queries halves it and moves the correlation from 0.45 to 0.64. Going from 400 to 1,600 halves it again and buys much less, because by then the confound and the proxy mismatch dominate. There is a point past which more judgments do nothing at all, and it arrives sooner than budgets assume.

Improve the proxy last, because it is the hardest. The alignment slider is the one with the most authority over the outcome and the least tractable mechanism. Moving it means choosing a metric that better matches what users do — weighting by dwell rather than clicks, evaluating at the cut-off you actually display, using judgments written against your product’s notion of a good answer rather than a generic one.

Where to put the threshold

Raising the bar trades recall for precision, and the exchange is steep. Push the threshold from +0.04% to +0.12% and precision rises from 50% to 70% while recall falls from 61% to 44%. Fewer experiments, better hit rate, more good ideas thrown away.

Which way to lean is not a statistical question. It depends on the price of an online test:

That is the real relationship between these two notes. Making online tests cheap is what lets you stop over-trusting the offline gate.

The artefact worth building

Almost nobody does this, and it takes an afternoon.

Keep a table with one row per experiment: what the offline metric predicted, what the online experiment measured, and when. After twenty experiments you can plot your own version of the chart above, using your own data instead of a simulation — and you will know your team’s actual offline–online correlation rather than assuming it.

Everything on this page then becomes concrete. You will know whether a 1% offline gain is worth an experiment slot. You will know which offline metric among the several you compute is the one that predicts. And you will be able to answer, in a review, the question that currently has no answer: how often has this been right before?