Twenty times less traffic to call the same winner

You have four candidate rankers and one production surface. Each A/B test needs two weeks and half your traffic, and the effects you are chasing are one or two percent. Four candidates, run properly, is a quarter.

Most teams respond by shipping on faith, or by trusting an offline metric that they know does not transfer. There is a third option that has been standard at search companies for twenty years and is still oddly rare everywhere else.

Why the A/B test is so expensive

Split your users, show half of them ranker A and half ranker B, and compare clicks per session. The comparison you are making is between two groups of people, and people are wildly unalike.

In a typical week, a small fraction of users generate most of the sessions. Someone who searches forty times contributes forty sessions of clicks; someone who searches once contributes one. When you randomly split users into two buckets, the heavy users do not divide evenly, and the resulting difference in clicks per session between the two buckets can easily exceed anything your ranking change did.

The sample size you need falls straight out of that:

The effect is in the denominator squared — halve the effect and you need four times the traffic. And in the numerator is user variance, which has nothing to do with your ranker at all. You are paying for traffic to average away a nuisance.

The idea: don’t split the users, split the list

Interleaving shows every user results from both rankers at once, mixed into one list, and works out which ranker’s contributions got clicked.

The standard construction is team-draft interleaving, and it is exactly the schoolyard procedure the name suggests.

The coin flip matters. Without it, whichever ranker always picked first would systematically get the better positions, and position bias would hand it the win regardless of quality. Alternating by coin flip makes the position distribution the same for both teams in expectation, which is what makes the comparison fair without needing to model examination at all.

Then count sessions: if of the sessions that expressed a preference went to B, test against a coin flip.

Run both

Below, the same two rankers are evaluated both ways over the same simulated traffic. The controls that matter are the size of the improvement and how different the users are.

At the defaults — a change worth about 4% in true nDCG@10, and realistic variation between users — the A/B test needs roughly 33,000 sessions and the interleaved experiment needs about 1,700. Same confidence, same conclusion, twenty times less traffic.

Where the twenty comes from

Drag how different users are down to zero, so every user clicks at the same underlying rate. The advantage collapses to under 7×. Drag it back up and the gap widens again.

That is the mechanism, and it is worth stating plainly: interleaving does not find a better signal. It removes a source of noise. The same user, in the same session, with the same mood and the same intent, is compared against themselves. Every property of that person — how much they click, how long they browse, how badly they wanted the thing — appears on both sides of the comparison and cancels.

The second saving is in the list. Look at the two rankings at the bottom of the widget: for a small ranking change, most of the ten results are the same documents in a slightly different order. Interleaving only ever learns from the slots the rankers disagree about, so the noise from the shared results — which in an A/B test is counted in full, on both sides — is never introduced.

Now drag the improvement down. At a 4% effect, the ratio is around 20×. At a 1% effect it is over 60×. The advantage grows exactly as the effect shrinks, which is the opposite of how experiment budgets usually behave and is why this technique earns its keep on mature systems where every remaining win is small.

What interleaving cannot do

This is not a general replacement for split testing, and treating it as one is the standard way to get burned.

• It only measures preference between rankings Revenue per user, subscription conversion, seven-day retention, session length — none of these can be attributed to one ranker inside a blended list. If your decision rests on a business metric, interleaving cannot make it for you.
• It measures clicks, and clicks are not satisfaction A ranker that wins on interleaved clicks may be winning on bait. The credit signal can be upgraded — weight by dwell time, count only clicks with no quick return to the results page — but the underlying limit remains.
• It cannot test anything that is not a list of results A new layout, a different snippet, a filter chip: there is no way to blend those into one page and attribute the outcome. Interleaving needs two orderings over the same kind of item.
• Every user is in the experiment There is no clean holdout — you are serving a mixture nobody would ever ship. If B is bad, every user gets some of B. Keep interleaved experiments short, and do not interleave a change you have not sanity-checked offline.
• Long-run effects are invisible Novelty, learned behaviour, the way a feed changes what people look for over months — a session-level preference test sees none of it.

The workflow that actually works

Use the two together, in the order that spends the least traffic on the least certain thing.

This is roughly what large search teams do, and the reason is not statistical elegance — it is that the interleaving stage is cheap enough to let people try ideas that would never have justified a two-week A/B slot.

Getting the implementation right

Run A/A first. Interleave production against itself. Team draft on two identical lists produces the same documents with the ownership alternating at random, so the preference must come out at 50% and the experiment must never reach significance. Set the improvement slider to zero in the widget to see what that looks like. If your harness finds a winner there, the credit attribution is wrong, and every result it has ever produced is suspect.

Do not peek. The sign test assumes a fixed sample size. Watching a preference rate wobble across 50% for a week and calling it the moment it crosses significance inflates the false positive rate badly. Either fix the sample size in advance, or use a sequential test designed for continuous monitoring.

Deduplicate carefully. Team draft’s “not already in the list” step is doing real work. If the same product appears under two ids, or two chunks of the same document both surface, one team can be credited for what is effectively the other’s result.

Watch the tie rate. Sessions with no clicks, or with equal credit, carry no information. If 60% of your sessions are ties, your effective sample is 40% of your traffic, and the readouts in the widget account for that. A high tie rate is usually a sign the two rankers agree on almost everything — which is itself worth knowing before you spend two weeks measuring it.

Consider probabilistic interleaving if you need reuse. Team draft is simple and unbiased, but the resulting log is hard to reuse for anything else. Probabilistic interleaving samples from each ranker’s distribution instead, which gives you an interleaved log you can evaluate a third ranker against offline. Start with team draft; reach for the probabilistic version when the offline reuse is worth the extra complexity.