The recommender eats itself

Here is the architecture almost every recommender ends up with, drawn honestly:

The model ranks the catalogue for a user.
The top few are shown. Everything else is not.
The user clicks something.
Those clicks — and only those clicks — become tomorrow’s training data.
Go to 1.

Step 2 is the problem, and it is not a bug in anyone’s code. It is what a recommender is. The consequence is that the model’s training data is not a sample of the catalogue. It is a sample of the model’s own past opinions, and every round it gets more confident about a smaller part of the world.

Thirty rounds

Below is a catalogue of 120 items across 8 topics, and 240 users with genuine preferences the system does not know. Round 1 is a random shuffle, because there is no evidence yet. After that the system is on its own: it ranks by what it believes, users click, beliefs update, repeat.

Press Run 30 rounds and watch the grid at the bottom.

Thirty rounds of a self-trained recommender. Purple is the share of the catalogue shown to anybody in that round; teal is the true quality of the match users received, as a share of the best possible; amber is the Gini coefficient of cumulative impressions. Each square in the grid is one item, brightest where it got the most impressions this round.

What the run shows

Start with greedy — the default, and what a system does if nobody has thought about this.

By round 5 the catalogue reachable in a single round has fallen from 120 items to 35. It never recovers. The Gini coefficient of impressions climbs past 0.8, meaning a small fraction of items is absorbing nearly all exposure. Seven of the twenty highest-quality items in the catalogue get no impressions at all — not “few”, none — and since they get no impressions, they generate no clicks, and since they generate no clicks, they will never be shown again.

The grid makes this visceral. It starts as an even wash and goes almost entirely dark, with a handful of cells burning bright.

Match quality rises for the first few rounds and then flattens at about 0.33 against a ceiling of 0.60. It does not crash. If the only number on your dashboard is click-through rate, this run looks like a system that learned quickly and then converged. Everything described above happened underneath a metric that was reporting success.

Why it happens without a popularity feature

There is no popularity term anywhere in this model. The collapse comes from two ordinary pieces of engineering meeting each other.

The estimate is smoothed. An item with 3 clicks from 30 impressions gets a click-rate estimate of . An item with no impressions at all gets the prior, . That smoothing is there for a good reason — without it, one lucky click on one impression would give an item a 100% click rate and it would take over the site. But it also means an unseen item is permanently rated below any item that has ever performed decently.

Selection is greedy. The system shows the top 5 by estimated score. An item rated 0.05 never enters a top 5 that contains items rated 0.08.

Put those together and the state is absorbing. An item’s estimate cannot improve without impressions; it cannot get impressions without a better estimate. What determined which items ended up on which side of that line was round 1 — a random shuffle. Luck, amplified thirty times.

Any system that (a) estimates value from observed outcomes, (b) shrinks unobserved things toward a fixed prior, and (c) selects greedily on the estimate will do this. It is not specific to recommenders. Hiring pipelines, credit models and content moderation queues all have the same shape.

The fix is exploration, and it is not a tax

The instinct is to add a diversity re-ranker, or a “boost new items” rule with a hand-tuned constant. Both help a little and neither addresses the cause, which is that the system has no way of representing not knowing.

Switch the widget to ε-greedy and set exploration to 25%. Coverage stays above 90%, Gini falls to 0.59, and no top-20 item is left unshown. Match quality lands at 0.335 — essentially identical to greedy’s 0.331. You gave away a quarter of your slots at random and the users did not notice.

Now switch to UCB and push exploration up. This is the interesting one. UCB does not explore at random; it adds a bonus that is large when an item has few impressions and shrinks as evidence arrives:

The bonus is an explicit statement of how unsure we are, and it makes the system show things because it does not know about them rather than in spite of not knowing. At : match quality 0.425, against greedy’s 0.331. That is 28% better for users, with 45% catalogue coverage instead of 29% and a markedly lower Gini.

Read that again, because it is the part people find surprising. The policy that spent slots on uncertain items delivered better recommendations than the one that always showed its best guess. Greedy was not being efficient. It was being early — it froze its opinion while that opinion was still mostly noise, and then spent twenty-nine rounds defending it.

ε-greedy

Trivial to implement, works, and wastes most of its exploration on items that were genuinely bad. Best when you care about catalogue coverage in its own right — marketplace supply, publisher relations, contractual exposure.

UCB / Thompson sampling

Spends its exploration where the uncertainty is, so most of it turns into information you can use. Best when the goal is the users’ outcome. Needs a per-item uncertainty estimate, which is the real implementation cost.

Why your offline evaluation will not catch this

This is the part that turns a known problem into a shipped one.

You hold out last month’s interactions, train two candidate policies, and measure which one predicts the held-out clicks better. The greedy policy wins — comfortably.

It wins because the held-out set was also produced by the loop. It contains interactions with the items the old policy chose to show, and no interactions at all with the items it suppressed. A policy that recommends a suppressed item gets no credit, because there is no record of anyone ever having a chance to click it. Offline, exploration is indistinguishable from error.

This is the same structure as position bias, one level up: there, the log under-counts documents that were shown low; here, it has nothing whatever about documents that were not shown. Inverse propensity weighting can rescue the first case. It cannot rescue the second — a propensity of zero has no inverse. Position bias is correctable arithmetic. Selection bias is missing data.

The practical consequence: the only honest evaluation of an exploration policy is online, and the only way to make the online test fair is to run it long enough that the exploration has had time to pay for itself. A one-week test of a policy whose whole thesis is compounding information will show you the cost and none of the benefit.

What to monitor

Click-through rate will not tell you this is happening. Put these next to it:

Catalogue coverage The share of items that received at least one impression in the last day and in the last week. It is the simplest number here and the fastest to move. A steady decline is the alarm.
Impression Gini, or the top-1% share How concentrated exposure is. Track it as a trend rather than against a target — what matters is the direction, because every recommender is somewhat concentrated and yours is not comparable to anyone else’s.
Time to first impression for new items How long a newly added item waits before anyone sees it. In a collapsed loop this quietly goes to infinity, and it is usually the first thing suppliers complain about.
Performance of items by age of first exposure If items discovered recently perform as well as long-established ones, your system is still finding things. If they perform much worse, either your exploration is badly targeted or you have run out of catalogue to discover.

Three things worth doing before any of this is a problem

Reserve slots rather than tuning a boost. “One of the five slots is drawn from a candidate pool of under-exposed items” is a rule that survives a model change, a re-platforming and a team handover. A +0.15 added to a score does not — the next person to touch the ranker will remove it, because nobody remembers what it was for.

Log the propensity at serve time. Whatever randomness you introduce, record the probability with which each item was chosen, next to the impression. It costs one float per row and it is the difference between being able to evaluate policies offline later and not. Nobody has ever regretted logging it; plenty of teams have spent a quarter reconstructing it.

Treat cold start as a permanent condition. New items arrive continuously. The system’s ability to give a brand-new item a fair hearing is not a launch-week concern, it is the mechanism that keeps the whole thing from slowly converging on the catalogue you had two years ago.