Notes
Working notes on ranking and personalization
Interactive explainers on the parts of a recommender that are hard to picture from a formula — hybrid retrieval, index compression, position bias, cold start, experiment design, calibration, privacy, and what a system trained on its own output does to a catalogue. Every widget on these pages runs the real algorithm in your browser.
-
An agent's memory is a user profile
Four hundred turns, twenty-four facts, and three ways to decide what to write down. The smallest memory answers best — because similarity search cannot tell a current fact from one that used to be true.
Read -
Private by arithmetic
Differential privacy costs almost nothing to count a population and almost everything to describe a person. The same ε, two very different bills — measured on the same profiles.
Read -
Personalizing without an account
Most visitors are anonymous and most sessions are short. How much of a logged-in experience can you rebuild from one visit — and how quickly? One event closes half the gap; eight close nearly all of it.
Read -
What the session is looking at
A single attention head trained in your browser on browsing sessions, against a bag-of-items average and a last-item heuristic — and the kind of dependency only one of the three can represent.
Read -
Features that know how the story ends
One aggregate run over the whole table, and a validation score two and a half points too high. Point-in-time correctness, training–serving skew, and why the damage is to your measurement rather than necessarily to your model.
Read -
Offline said it would work
Offline evaluation is a screening test, and screening tests have a precision. At realistic settings half of what clears the bar is a genuine win, a quarter actively hurts — and the gate is still worth keeping.
Read -
How deep should the expensive stage look?
Reranking more candidates costs latency and money, and past a point it also costs quality — because every extra candidate is another chance for an imperfect reranker to promote something that does not belong.
Read -
A score is not a probability
A model with an AUC of 0.87 that forecasts 32% more conversions than happen. Reliability diagrams, Platt scaling and isotonic regression, and the moment the number stops being sorted and starts being spent.
Read -
What are you actually ranking for?
Clicks, dwell time and margin pulling in different directions, a weight for each, and the Pareto frontier that shows which trades exist — including the one that buys 37% more revenue for 8% fewer clicks.
Read -
Giving items an id that means something
Residual quantization turns an embedding into a short sequence of tokens, so similar products end up with similar ids — and recommendation becomes generating the next id rather than searching for it.
Read -
Two towers, and the index underneath them
A two-tower retriever trained in your browser with in-batch negatives, then served through an IVF index — where the only knob you get at query time trades recall against how much of the catalogue you touch.
Read -
The recommender eats itself
Thirty rounds of a system trained only on what it chose to show. The catalogue collapses from 120 items to 35, seven of the best twenty disappear entirely — and the click-through rate barely moves, which is why nobody notices.
Read -
Twenty times less traffic to call the same winner
An A/B test and a team-draft interleaving experiment run against the same pair of rankers, with the sample size each one needs to reach significance — and the reason the gap is almost entirely about how different your users are from each other.
Read -
Clicks are not relevance
Train a ranker on its own click log and you mostly re-learn the ranker you already had. Position bias, the inverse-propensity fix, and the variance that fix brings with it — simulated live.
Read -
Shrinking a vector index by 32×
int8, binary and Matryoshka compression applied to the same embeddings, scored two ways — and the moment where recall says you broke search while the users' metric says nothing happened.
Read -
Diversity without wrecking relevance
Maximal marginal relevance with a λ you can drag — the point where six nearly identical results become six useful ones, and the point just past it where the list stops answering the question.
Read -
Reading a ranking metric: NDCG, MRR and the rest
Precision, recall, MRR, average precision and NDCG computed live from a list you can reorder and regrade — including the reorderings that move one metric and leave another completely still.
Read -
Fusing two rankers without calibrating their scores
Reciprocal rank fusion, weighted RRF and CombSUM, compared in a playground where a keyword ranker and a concept ranker disagree — and you decide how to reconcile them.
Read -
Five recommender algorithms, one catalog
Popularity, item-item CF, content-based, BPR matrix factorization and a hybrid, all ranking the same synthetic catalog for the same user — including what happens when the user is brand new.
Read -
How fast should a user profile forget?
Exponential decay on a user profile, a half-life you can drag, and the moment where four days of debugging Kubernetes buries a year of interest in retrieval.
Read