What the session is looking at

A user profile is usually a summary of a history: sum the things they touched, maybe weight the recent ones more, normalise, done. It is cheap, it updates incrementally, and it is what most production systems still run.

It also throws away the order, and the order is frequently the whole message.

Two sessions, same items

Consider a session containing: a camera body, a lens cap, a tripod, a memory card. Now the same four items in a different order, ending on the camera body.

A bag-of-items profile produces the identical vector for both. But the first user is accessorising a camera they have decided on, and the second is still choosing the camera. Those want different recommendations, and no amount of tuning the weights of an average will separate them, because the average is the same object.

What attention actually adds

The mechanism is worth stating precisely, because “attention” gets used as a synonym for “modern” and it is doing something specific.

Take the items in a session, each embedded as a vector . A fixed summary computes for weights that do not depend on the content — an average uses , a decayed profile uses .

Attention computes the weights from the session itself:

The query comes from where the user is right now; the keys come from what they did earlier. The dot product is a learned notion of “is this earlier item relevant to what is happening now”, and the softmax turns it into a weighting.

Watch three models learn the difference

Below, a catalogue of 42 items across six aisles. Each aisle has an anchor body and a set of accessories, and every anchor has one accessory it is bought with.

A session wanders through several aisles, picking up two or three anchors along the way. The last item says which aisle the user is in now. If that aisle’s anchor turned up earlier, the next thing they want is its matching accessory. If it did not, they are still shopping for the body itself.

Three models are trained on the same sessions with the same code, differing only in what they may attend to. They fit in your browser — the loss curves fill in as they go.

Reading the result

Self-attention: 87% next-item accuracy. Bag of items: 64%. Last item only: 55%.

The last-item model knows which aisle the user is in and nothing else, so it has to guess between two specific items and lands near the coin flip that implies.

The bag-of-items model is the interesting failure. It sees every anchor in the session — they are all in the average. What it cannot do is work out whether the one that matters is among them, because that requires combining “which aisle are we in” with “which anchors are present”, and both facts are already mashed into one vector. It does better than chance by learning which aisles have common anchors, and it plateaus there.

Now switch to anchor near the end and back to anchor at the start, and watch the attention bars. The weight follows the anchor around the session. It is not recency, and it is not uniform. It is a lookup.

Reading attention weights is a debugging tool

That last observation is the practical payoff, and it is underrated. When a sequential model recommends something strange, the attention weights tell you which part of the session it blamed. On a real system this catches:

• Contamination A shared device, a gift purchase, a research rabbit hole. If the model is attending hard to three items from a session that was not this person, you can see it — and you have a mechanism for down-weighting them that a summed profile does not offer.
• Position collapse If attention always lands on the last position regardless of content, the model has learned nothing the last-item heuristic does not already know, and you are paying transformer inference for it.
• Attention sinks Real models frequently dump weight on the first token or on a common item, essentially as a null option. This is normal and worth recognising rather than debugging — but if the sink absorbs most of the mass, the head is not contributing.

What this costs in production

Sessions have to be defined. Thirty minutes of inactivity? A day? Until the app is backgrounded? The model’s notion of “now” is whatever you decide here, and the decision changes the data more than most hyperparameters do.

Length is a hard cap. SASRec-style models take the last items, and anything older is invisible. That makes them a short-term model by construction — which is why the mature pattern is a sequence model for immediate intent alongside a slow profile for settled taste, blended the way the two-half-life profile is.

Position needs encoding, and the choice matters. Ordinal positions work when sessions are dense. When gaps between items vary from seconds to weeks, the useful signal is elapsed time, not index — and a model given only indices will treat a fortnight’s gap as one step.

Serving is heavier. A profile vector is a lookup. A sequence model is a forward pass over the session at request time, and it cannot be precomputed because the session is what changed. In a two-stage stack that puts it in the ranking stage, not candidate generation — or in candidate generation only if you can encode the session into a vector fast enough for a nearest-neighbour lookup, which is what the two-tower version of this architecture is for.

When not to bother

If your users’ taste is stable and their sessions are short, a decayed profile will get most of the value for a fraction of the complexity. The gap between the attention model and the bag of items in this widget is large because the catalogue was built with a dependency that needs binding — many catalogues do not have one that strong.

The honest test is cheap: take your logs, build both profiles, and check whether next-item prediction improves. If the sequence model beats the bag by two points, the order was not carrying much, and you have learned something worth knowing before committing to serving a transformer on every request.