Giving items an id that means something

Every recommender has to refer to items somehow, and almost all of them use a number that means nothing: product_88213. The number is arbitrary. Item 88213 and item 88214 have no more in common than any other pair.

That single arbitrary choice has consequences that run through the whole system.

Memory is linear in catalogue size If an id means nothing, the model has to learn what it means, which requires an embedding per id. Ten million items at 128 dimensions is a five-gigabyte embedding table, and it is usually the largest thing in the model by an order of magnitude.
New items start at zero An id the model has never seen has no row in that table. Whatever the item is, however obviously similar it is to something already selling well, the model has nothing to say about it until it has accumulated interactions.
Nothing generalises across items Learning that people who buy one Scandinavian desk lamp buy another tells the model nothing about a third one, because all three ids are unrelated tokens.

The idea behind semantic IDs is to stop using an arbitrary number and start using a short code derived from what the item actually is.

Residual quantization, plainly

Start with an embedding — from a content encoder, a text model, whatever produced a vector that puts similar items near each other. The goal is to replace that vector with a handful of small integers, without losing the “similar items are near each other” property.

Run k-means over every item embedding with a small number of centroids — say eight. Each item’s first token is which centroid it landed nearest.
Subtract that centroid from the item’s vector. What is left is the residual — everything the first token failed to capture.
Run k-means again, this time over the residuals. Each item’s second token is which of those centroids its residual landed nearest.
Subtract again, and repeat for as many levels as you want tokens.

The result is a tuple like . Because each level only ever describes what the previous levels missed, the code reads coarse to fine from left to right: the first token says roughly what kind of thing this is, and the last tokens say which particular one.

That ordering is the whole point. Two items sharing a first token are in the same broad neighbourhood. Two items sharing the first two are closer. The id has become a path down a tree rather than a label.

Build some

Below, 420 items with 32-dimensional embeddings get real residual quantization — k-means over the vectors, then k-means over the residuals, once per level.

How much structure survives the compression. Purple is the share of an item's ten true nearest neighbours that share its whole code prefix; teal is how much of the original vector the code still fails to explain. Below, one item's code and its nearest neighbours under the same first token, in order of true similarity — read the codes down the column against the similarities on the right.

The trade, in two numbers

At the default — an 8-way codebook, two levels — each item is described by 1.5 bytes instead of a 128-byte vector, and 82% of every item’s true neighbourhood still shares its full prefix. That is a remarkable amount of structure to survive six bits.

But look at the third readout: there are only 32 distinct prefixes for 420 items. The code is not an identifier. It is an address for a neighbourhood, and about thirteen items live at each address.

Now push the codebook to 32. Suddenly there are 326 distinct prefixes and almost every item has its own — and neighbourhood purity has collapsed to 4%. The code has become a unique identifier again, which is exactly what it was supposed to stop being.

A code that identifies items uniquely carries no similarity information. A code that groups similar items cannot address them individually. You do not get to resolve this by tuning — you resolve it by accepting both: a semantic prefix that groups, plus a final token that disambiguates. That last token is the “+4 bits” in the byte count, and it is why the published schemes all have one.

Watch the teal line as you add levels. Each level cuts the unexplained residual further, but by less than the one before — the first token does the most work, and by the fourth the returns are visibly diminishing. That shape is the reason these schemes use three or four levels and not twelve.

The part that makes this more than compression

Once an item is a sequence of tokens, recommending it becomes generating that sequence.

Feed a model the tokens of the user’s last several interactions, and train it to emit the tokens of the next item — a sequence-to-sequence task, the same shape as translation. At serving time, run beam search: pick the most likely first token, then the most likely second given the first, and so on. Whatever full code you decode is the recommendation.

There is no vector index in that sentence. No ANN, no nprobe, no recall against exhaustive search, none of the machinery that normally sits under a retriever. The model’s output layer is the size of the codebook — eight, or 256 — instead of the size of the catalogue.

What that buys

The embedding table stops scaling with the catalogue. A new item gets a code the moment its content is encoded — no interactions required. And the model learns across the hierarchy: evidence about one item genuinely informs its prefix siblings, which is the generalisation an arbitrary id could never provide.

What it costs

Decoding is autoregressive — three or four sequential model calls per request, against one lookup for a vector index. Beam search can emit codes that do not correspond to any item, so decoding has to be constrained to a trie of valid codes. And diversity in the output has to be engineered, because beam search is drawn to the same high-probability prefix.

The thing that will actually bite you

Every operational problem with this approach is a version of the same one: the ids are derived, so they move.

Retrain the content encoder and every embedding shifts slightly. Re-run k-means and the centroids land somewhere else. Item 88213, which was , is now . Nothing about the product changed, and its identifier did.

That has consequences a normal id never has:

Every model downstream is invalidated. The sequence model was trained on the old vocabulary. Its learned associations are about codes that no longer point where they did. This is not a fine-tune; it is a retrain.

Your logs become bilingual. Interactions recorded before the recoding use the old codes. Any training run spanning the change needs a mapping, and any analysis that groups by code silently mixes two different things.

Caches and A/B assignments break. Anything keyed on the id — a precomputed slate, a bucketing hash, a feature store row — is now keyed on something else.

The standard mitigation is to keep the stable numeric id as the system of record and treat the semantic code as a derived index key, versioned explicitly, with a mapping table maintained for at least as long as your training window. It works and it is more machinery than it first sounds like.

Would I ship this?

Honestly: it depends on the catalogue, and I would not present it as the obvious successor to a two-tower retriever.

Worth a serious look when the catalogue turns over quickly and cold start is the dominant cost — marketplaces where sellers add items continuously, media platforms with a large daily upload, anything where a meaningful share of impressions should go to items that are days old. The ability to place a brand-new item into a neighbourhood before it has a single interaction is a real advantage, and it is not one that classical collaborative filtering can match.

Probably not yet if the catalogue is stable and large and your existing retrieval works. The published gains over a well-tuned two-tower model are respectable rather than transformative, the serving path is more complex, and the renumbering problem is a permanent operational tax.

Worth doing regardless of any of the above: build the codes and look at them. Residual quantization over your item embeddings takes an afternoon, and the resulting tree is the best map of your catalogue you will ever get for that effort. Whether or not you ever generate from it, knowing what your encoder thinks the coarse structure of your inventory is tends to be worth the afternoon on its own.