Clicks are not relevance

You have a search ranker in production, a year of click logs, and an obvious idea: train the next ranker on what people clicked. The data is free, it is enormous, and it is exactly the behaviour you are trying to optimise.

The new model trains. Offline it looks excellent — it predicts the held-out clicks well. You ship it, and nothing happens. No lift, no regression, no signal. You have spent a quarter rebuilding the ranker you already had.

This is the single most common way a ranking project quietly fails, and it has a precise cause.

A click is two events, not one

Nobody clicks a result they did not look at. So a click is the conjunction of two separate things:

Only the second factor is about the document. The first is about where the document happened to sit on the page — and it varies enormously. Eye-tracking studies, and every click log anyone has ever plotted, find the same steep decay from the first result down. A common way to write it is:

with somewhere near 1 on a typical search page — meaning the tenth result is examined about a tenth as often as the first. On a mobile feed it is steeper. On an infinite scroll it is steeper still.

This is called the position-based model, and although it is a simplification — it ignores that people stop after they find what they wanted, and that a good result above yours makes yours less likely to be examined — it captures the dominant effect, and it has the great virtue that the bias term depends only on something you know exactly: where you put the document.

If clicks were relevance plus random noise, more data would fix it. But the distortion is not random — it is a systematic function of the ranking you were already serving. More data makes the estimate of the wrong quantity more precise. This is why the offline metric keeps improving while the online metric does not move.

Watch it happen

Below is one query with forty candidate documents. Each has a true relevance the simulation knows and you do not. A mediocre production ranker orders them, users arrive, examine positions according to , and click. Then the clicks are turned back into a relevance estimate and used to build a new ranking, which is scored against the relevance the simulation actually used.

One query, forty documents, clicks sampled from the position-based model. Each dot is a document: horizontally its true relevance, vertically what the estimator thinks. A perfect estimator would put every dot on the dashed diagonal. Start with raw clicks at η = 1.

Reading the damage

With raw clicks at and ten thousand sessions, the ranker you learn scores about 0.80 on nDCG@10. The ranker that generated the clicks scores 0.68. So the naive approach did improve things — which is precisely why this mistake survives code review. It is not catastrophic. It is a fraction of what was available.

The number to look at is agreement with the old ranker. Perfect learning would score 0.26 against the logging policy, because that is how much the old ranker happened to have right. Raw clicks score around 0.71. Most of what the new model learned is not relevance. It is the old model’s opinion, laundered through a click log and returned to you as evidence.

Now look at the scatter. The dots do not lie on the diagonal — they lie on several parallel diagonals, one per region of the page. Two documents of identical relevance get wildly different estimates depending on where they were shown, and no amount of extra data closes that gap.

Push to 1.5 and the learned ranker falls to about 0.74, barely above the policy that logged the data. Push it to 0 — a world where every position gets equal attention — and raw clicks are suddenly perfect. The problem was never the clicks. It was where they were collected.

The fix is one weight

If a document at rank is only examined with probability , then each click it received stands in for roughly clicks that would have happened had everyone looked. So count it that way.

For every logged impression, record the rank the document was shown at.
Take its propensity from the click model.
Every click contributes instead of 1; every non-click contributes 0.
Train — or simply estimate relevance — on those weighted counts.

Formally, the naive estimator converges to the wrong thing and the weighted one converges to the right thing:

Switch the widget to IPS. The dots collapse onto the diagonal, nDCG@10 goes to essentially 1.0, and agreement with the old ranker drops to 0.26 — exactly where perfect learning should sit. The clicks did contain the relevance signal all along. They were just being read at the wrong scale.

This is survey weighting. If you poll a thousand people and 90% of the respondents are from one city, you do not conclude the country agrees with that city — you re-weight each response by how likely that person was to have been sampled. A click log is a survey whose sampling probability is the position you chose.

The bill for the fix: variance

IPS is unbiased. Unbiased is not the same as reliable.

The weights are , and gets small down the page. At , a document at rank 30 has a propensity near 0.006, so a single click on it counts for about 160. One accidental click, from one user, moves that document a long way up your training signal.

Drag the sessions slider down to 500, set to 1.5, and press Re-run a few times. Raw clicks barely move between runs — it is biased, but it is stable. IPS jumps around by several points of nDCG, because a handful of high-weight clicks landed differently. Averaged over many runs, IPS is right. Any single run might not be.

The standard response is to stop the weights from getting arbitrarily large:

Switch to IPS, clipped. At 500 sessions it is the best of the three and the steadiest. Now raise sessions to 5,000 with at 1.5 and watch it lose: 0.80 against plain IPS’s 0.98. The clip has stopped correcting the very positions that needed correcting most.

Low traffic

Few clicks, and the rare deep click carries an enormous weight. Variance dominates. Clip hard, accept the bias, ship something stable.

High traffic

Enough clicks at every position that the weighted average is well estimated. Bias dominates. Loosen the clip, or drop it, and take the unbiased answer.

That is the whole bias–variance trade, on a dial you set yourself. Set it by holding out queries and measuring, not by copying a threshold from a paper.

Where do the propensities come from?

Everything above assumed you know . You do not, and this is the part that makes counterfactual learning-to-rank an engineering project rather than a formula.

Randomisation is the honest answer. If you occasionally swap two adjacent results at random, the same document lands at both ranks for comparable traffic, and the ratio of its click rates estimates the ratio of the propensities directly. A small amount of deliberate randomness — on a fraction of traffic, on adjacent pairs only — buys you a propensity curve you can defend.

Intervention harvesting is the cheap answer. You already ship ranking changes. Every one of them put some document at two different ranks for two different populations. Mining historical A/B tests for those natural experiments gives you propensities without adding any randomness at all, at the cost of a much fiddlier pipeline.

Estimate them jointly, with care. There are methods that learn the propensity model and the relevance model together from the same clicks. They work, and they are also the easiest way to get a confidently wrong answer, because the two factors are only separable given assumptions that no log can verify.

Underestimating the propensity at deep ranks inflates those weights and lets noise dominate. Overestimating it under-corrects and leaves you closer to the naive estimator. Of the two, over-correction is the more dangerous, because it looks like a bold new ranker rather than a timid one.

What position bias is not

Correcting for position does not correct for not being shown at all. In the widget every document is on the page somewhere, so every document has a propensity. In production, results below the fold — or below rank 50, or on page four — get essentially no impressions, and a document with zero impressions carries no information at any weighting.

That is selection bias, and it is a different animal: it needs exploration, not re-weighting. Something has to occasionally show documents the current ranker does not believe in, or their relevance is permanently unknowable and the catalogue slowly narrows to whatever the first model liked. That narrowing has its own dynamics, and they are worth watching separately.

Two more things the position-based model deliberately ignores:

Users stop when they are satisfied Someone who finds their answer at rank 2 never examines rank 6. The cascade family of click models handles this by making examination depend on what happened above, at the cost of a much harder estimation problem.
Attractiveness is not relevance A click responds to the title, the thumbnail and the price — everything visible before the click. Whether the document was actually any good shows up after it, in dwell time, in whether they came back to the results page, in whether they bought. Optimising clicks alone reliably produces a ranker that is very good at bait.

What I would do on a real system

Start by measuring rather than assuming it. A week of small adjacent-pair randomisation on 1% of traffic is usually enough, and the resulting curve is worth having regardless of what you do next — it tells you how much of your click-through rate is a property of your layout rather than your ranking.

Then correct, clip generously at first, and — this matters — evaluate the corrected model with a corrected metric. An IPS-trained ranker scored on raw click-through will look worse than the naive one, because the naive one was optimising exactly that biased quantity. If you fix the training signal and leave the evaluation biased, the correct model loses the review and never ships.