Evaluation
Reading a ranking metric: NDCG, MRR and the rest
Precision, recall, MRR, average precision and NDCG computed live from a list you can reorder and regrade — including the reorderings that move one metric and leave another completely still.
Someone asks whether your new ranker is better. You run the evaluation and it comes back: NDCG@10 up 1.2%, MRR flat, recall unchanged.
Knowing what that sentence means — which of those numbers could have moved, which could not, and which one you should have been watching — is most of what ranking evaluation is.
The fastest way to learn it is to break the metrics on purpose.
What a ranking metric is actually doing
Start from the problem, because the formulas make much more sense once the shape of the task is clear.
You have a query. You have a list of results, in the order your system chose. Somebody — a human assessor, or a proxy like a click log — has said how good each result is. A ranking metric is a function that turns those two things into one number, so that “is A better than B?” becomes arithmetic.
Every such function has to make three decisions, and every metric on this page is just a different set of answers:
- How far down do we look?
Nearly all of them stop at a cut-off, written
@k. Results below position k are treated as if they do not exist. If your product shows five results, a metric measured at 10 is scoring things nobody saw. - Is relevance yes/no, or graded? Precision, recall and MRR ask only “is this relevant at all”. NDCG reads a scale — not relevant, marginal, relevant, perfect — and treats a perfect result as worth far more than a marginal one.
- Does position inside the window matter? Precision and recall say no: the same five documents in any order score the same. NDCG says yes, and applies a discount that grows with position.
The reason there is no single right metric is that these are product questions, not statistical ones. A metric that ignores order is exactly right for a retrieval stage feeding a reranker, and exactly wrong for the final list.
The list
Ten results for the query “how do I combine two rankers”, each with a graded relevance judgment from an assessor: not relevant, marginal, relevant, perfect.
Reorder the list with the buttons. Click any judgment to change it. Drag the cut-off. Everything recomputes from the list as it actually stands.
Five metrics over one list. The bar on each row is that position's discounted gain — the amount that row contributes to DCG. Everything below the cut-off is dimmed but still shown, because seeing what a metric ignores is most of the point.
The curve at the top plots all three against every cut-off at once, and the shapes are the lesson on their own. Recall only ever climbs — a bigger window can only find more. Precision usually falls, because the good results are near the front. NDCG does neither reliably, because it is the only one of the three reading position and grade together.
What each one can and cannot see
Precision@k — blind to order
The share of the top that is relevant. It has no idea what order those are in.
Try it: set and hit Reverse. Precision@5 will very likely be unchanged, because reversing the whole list mostly swaps items between the top five and the bottom five — but hit Worst first and it collapses. Precision notices what is in the window, never where in the window.
Recall@k — depends on the pool
The share of everything relevant that you managed to surface. Note the denominator: it counts relevant documents in the judged pool.
That makes recall treacherous to average across queries. A query with two relevant documents and a query with forty are being scored on completely different scales, and the mean of the two means very little. It is also why recall is the natural metric for the retrieval stage — where the job is literally “do not lose the good ones” — and a poor one for the final ranking.
MRR — one result deep
The reciprocal rank of the first relevant result, averaged over queries.
Try it: grade the top result “not relevant” and watch MRR halve. Now change anything at all below position two — MRR does not move at all.
That is not a flaw; it is the metric’s opinion. For known-item search — someone looking for one specific thing they know exists — the rest of the list genuinely does not matter. For a browsing feed it is the wrong metric entirely.
MRR and precision both collapse graded judgments to a binary. If your assessors carefully distinguished “perfect” from “marginal”, these two metrics throw that work away.
NDCG — graded, and position-aware
The one that reads both dimensions. Gain from the grade, discounted by position:
Then normalize by the best ordering possible with those same judgments:
Two design decisions are doing the work.
The exponential gain, , means a “perfect” result is worth 7 while “marginal” is worth 1. Grades are not treated as a linear scale — one excellent result outweighs several mediocre ones.
The logarithmic discount is much gentler than people assume. Position 10 is divided by , so it still contributes about 29% of what position 1 would. If your product only ever shows three results, NDCG@10 is generously crediting you for seven you never displayed.
The normalization is what makes it comparable across queries: a query where the best achievable DCG is small gets scored against that ceiling, not against a query with ten perfect documents in the pool.
NDCG@5, worked all the way through
Take five results with grades 3, 1, 2, 0, 3 — a perfect answer first, then a marginal one, then a decent one, a dud, and another perfect one.
| Position | Grade | Gain | Discount | Contribution |
|---|---|---|---|---|
| 1 | 3 | 7 | 1.000 | 7.000 |
| 2 | 1 | 1 | 1.585 | 0.631 |
| 3 | 2 | 3 | 2.000 | 1.500 |
| 4 | 0 | 0 | 2.322 | 0.000 |
| 5 | 3 | 7 | 2.585 | 2.708 |
| DCG@5 | 11.839 |
Now the ideal: the same five judgments in their best possible order, 3, 3, 2, 1, 0.
Three things are worth reading off that table.
The first position contributes 59% of the total DCG on its own. The discount is gentle — the same document at position 5 would still be worth , or 39% of what it earns at position 1 — but combined with the exponential gain it means the top slot dominates.
Grade outranks position, and it is not close. The perfect result sitting at position 5 contributes 2.708; the marginal one at position 1 contributes 0.631. Moving a mediocre document to the top is worth far less than finding a better one to put anywhere in the window.
And the document graded 0 contributes exactly nothing, which is why adding an irrelevant result to the bottom of a list can never lower NDCG. It can only fail to raise it — so a system that pads its results costs you nothing on this metric and a great deal with users.
IDCG is computed from the judgments you have. If your assessors only judged the top 10 results of the old system, and the new system surfaces a brilliant document nobody ever judged, that document scores as grade 0 — and your new ranker is penalised for finding something. This is the single most common way an offline ranking evaluation lies to you, and the fix is to pool the candidates from both systems before sending anything for judgment.
Average precision — the compromise
Precision recomputed at every position where a relevant document appears, then averaged. It is order-aware like NDCG and binary like precision — a reasonable default when your judgments are binary anyway.
The experiment worth running
Set so nothing is hidden, then click Ideal order and note every number. Now swap just the top two results by hitting Reverse twice on a shuffled list — or simply regrade to force a swap.
- Precision@10: identical. Same items in the window.
- Recall@10: identical. Same items found.
- MRR: changes only if the swap moved the first relevant result.
- NDCG@10: changes, slightly.
Four metrics, one change, one number moving. If you are reporting the wrong one, a genuine ranking improvement is invisible — and so is a genuine regression.
Is 1.2% even real?
A number moved. Before anyone ships anything, there is a second question, and it is the one most offline evaluations skip entirely.
NDCG is computed per query and then averaged, so it is a sample mean, and it has a standard error like any other. With queries and a per-query standard deviation , the uncertainty on that average is:
Per-query NDCG is wildly variable — a around 0.25 is typical, because some queries are easy and some are hopeless. On a 200-query judged set, that is a standard error of about 0.018 on a metric whose absolute value might be 0.45. A 1.2% relative improvement is roughly 0.005 absolute. It is a quarter of the noise.
Two things make this tractable:
Pair the comparison. You are running both systems over the same queries, so test the per-query differences, not the two means. Query difficulty cancels exactly, and the standard deviation of the differences is usually four or five times smaller than the standard deviation of the scores. This is the same variance-reduction argument that makes interleaving beat A/B testing online.
Report the win/loss breakdown. “Improved on 84 queries, unchanged on 61, regressed on 55” is a far more useful sentence than a mean, and it is the one that leads to the queries worth actually looking at.
What I would actually report
One metric as the decision variable, chosen to match the decision the user is making. Known-item lookup: MRR. Browsing feed with graded judgments: NDCG at the number of results you actually display. Retrieval stage feeding a reranker: recall at the candidate-set size.
The others as guardrails, so you notice when the headline moved for a stupid reason.
A distribution, not a mean. “NDCG@10 improved 1.2%” hides the case where 80% of queries got slightly worse and 20% got dramatically better. Report the mean with the spread, and look at the queries that regressed. That list is usually more informative than the average.
And an honest note about what offline evaluation is for. It is a filter, not a decision. It is cheap, it is repeatable, and it catches disasters before they reach anybody. It cannot tell you whether users are better off, because the judgments were written by someone who was not the user, about a list that was assembled by the system you are trying to replace. Everything on this page earns its keep by deciding what is worth putting in front of real traffic — and nothing on this page can substitute for having done so.