Two towers, and the index underneath them

You cannot score ten million items with a neural network on every request. That one constraint is the reason modern retrieval looks the way it does, and the two-tower model is the shape the constraint forces.

The architecture, and why it is that shape

Put the user on one side and the item on the other, and never let the two meet until the very last operation:

An item tower maps each item’s features to a vector. This runs offline, over the whole catalogue, on a schedule.
The resulting vectors go into a nearest-neighbour index.
A user tower maps the request — history, context, recent session — to a vector in the same space. This runs once per request.
Retrieval is a nearest-neighbour lookup in that index. Nothing else is evaluated at request time.

The single design decision is that the two towers share no computation. The score is a dot product and nothing else:

That is a severe restriction. A model that could look at the user and the item together would be far more accurate — it could learn that this particular user cares about this particular attribute in this particular context. The two-tower model cannot express any of that.

It buys one thing in exchange, and the thing is decisive: because the item side does not depend on the request, every item vector can be computed in advance. Retrieval stops being “run a model ten million times” and becomes “look up a point in an index”.

Two-tower models are for candidate generation — millions down to hundreds. The accuracy you gave up is recovered in the next stage, where a cross-encoder or a gradient-boosted ranker scores those few hundred with all the interaction features you like. Nobody uses a two-tower model as their final ranker, and nobody uses a cross-encoder over a whole catalogue.

Training it: where do the negatives come from?

The model needs to learn “this user goes with this item”. You have millions of positive pairs — every interaction is one. You have no negatives at all, because nobody logs the items a user did not interact with, and the vast majority of those they simply never saw.

Sampling random items as negatives works, and it is wasteful: a random item is usually so obviously wrong that the model learns nothing from it after the first few epochs.

The standard answer is elegant enough that it is worth stating plainly. In-batch softmax: take a batch of real (user, item) pairs. For each user in the batch, treat their own item as the positive and the other items in the batch as negatives.

Nothing extra was sampled, nothing extra was encoded. The negatives are free, and there are as many of them as your batch is large — which is why two-tower training runs use batch sizes in the thousands, and why the batch size is a model quality parameter rather than a memory-management one.

In-batch negatives are sampled from the interaction log, so popular items appear as negatives far more often than rare ones — and the model learns to push them down. The standard correction is logQ correction: subtract $\log(\text{estimated sampling probability})$ from each logit, so a popular item being a negative counts for less. Without it, a two-tower model quietly under-retrieves your best-selling products.

Train one, then serve it

Below, both towers are linear maps over tag features, trained with in-batch softmax on roughly 3,500 synthetic interactions. It runs on page load in about 200 milliseconds and reports the wall clock.

Then the learned item vectors go into an IVF index: run k-means over the catalogue, and at query time compare the query against the cluster centroids first, then scan only the items inside the nearest few clusters.

The operating curve of an ANN index. Horizontally, how many vectors a query compares against; vertically, the share of exhaustive search's top-10 it still finds. The highlighted curve is the cluster count you have selected; the faint ones are the other three. The lower chart is the training loss — note that it converges well below ln 48 = 3.87, the loss of guessing at random among the batch.

Reading the operating curve

The default — 32 clusters, 3 probed — finds 90% of the exact top-10 while comparing about 89 vectors instead of 600. That is a 6.7× reduction in work for one result in ten going missing.

Drag the probe slider right. At 8 of 32 clusters recall is 100% and the speed-up is 3.4×. Drag it to 1 and recall falls to 70% while the speed-up rises to 10×. There is no setting that is simply better; there is a curve, and where you sit on it is a product decision about what a missing result costs.

Now the counter-intuitive part. Switch to 128 clusters and set probes to 1. Recall is 54% — worse than 16 clusters at one probe — and the number of vectors compared is 137, most of which are centroids. With more, smaller clusters, the answer is more likely to be in a cluster you did not probe, and you pay 128 comparisons before scanning a single item.

Minimising that over at fixed recall gives the familiar rule of thumb that the cluster count should be around — for 600 items, about 24, which is why the 32-cluster curve dominates in this widget. For ten million items it is a few thousand.

IVF

Cheap to build, trivially explainable, and the probe count is a live knob you can turn per query — spend more on a high-value request, less on a prefetch. Needs retraining when the distribution shifts.

HNSW

A navigable graph rather than clusters. Better recall at the same work in almost every benchmark, and the equivalent knob is ef_search. Costs several times more memory per vector and is slower to build, which starts to matter at exactly the scale where you also want to compress the vectors.

What the model is worth, separately from the index

The model quality readout answers a different question from recall: of the items exhaustive search returns, how many are genuinely in that user’s top decile by true affinity? Around 78%, and no index setting changes it. Recall measures the index against the model. That number measures the model against reality.

Keeping the two apart is the practical discipline here. A retrieval system that is disappointing has one of two problems, and they have nothing to do with each other:

The index is losing the model’s answers Recall against exhaustive search is low. Fix it with probes, cluster count, or a different index. It is an engineering problem with a dial on it.
The model’s answers are not good Recall is 100% and users are still unhappy. No index change will help. Fix features, negatives, the loss, or the freshness of the training data.

Teams that do not measure recall-against-exhaustive spend months tuning the wrong one of these.

Things that bite in production

The two towers must not drift apart. The user vectors your service computes today have to live in the same space as the item vectors your batch job wrote last night. Ship both towers as one artefact, versioned together, and make the index carry the version of the tower that produced it. A mismatched pair does not crash — it returns confident nonsense.

Item vectors go stale in a way user vectors do not. The user tower runs on every request and sees the session. The item tower ran when the item was last indexed. If your items change — price, availability, title, popularity signals — the index is a snapshot of yesterday’s catalogue. Decide the refresh cadence deliberately, and keep a fast path for items that change materially.

Filters and ANN fight each other. “Nearest neighbours, but only items in stock in this country” cannot be answered by probing clusters, because the filter is orthogonal to the geometry. Pre-filtering means maintaining an index per filter combination; post-filtering means over-fetching by whatever factor the filter removes, and hoping. This is the single most common reason a vector search deployment ends up slower than the keyword system it replaced.

Cold items work; cold users are still hard. Because the item tower reads features rather than an id, a new item gets a usable vector the moment it is encoded — a genuine advantage over classical matrix factorization, and the reason this architecture won. The user side has the same property only if your user features are contextual rather than historical. A brand-new user with no history still produces the same vector as every other brand-new user, and that is a decision somebody has to make explicitly.