A score is not a probability

Your conversion model has an AUC of 0.87. It ranks well, it beats the previous one on every offline metric, and it ships.

Two weeks later someone in finance asks how many conversions to expect from next month’s traffic. You add up the predictions and hand over a number. The month ends 32% below it.

Nothing about the model is broken. It was never asked to be right about magnitudes, and nothing you measured would have noticed if it was not.

Two jobs, one number

A model that outputs a number is being asked to do two separable things.

Discrimination

Does it put the events that will happen above the events that will not? This is what AUC measures, what NDCG measures, what every ranking metric measures. It depends only on the order.

Calibration

When it says 0.30, does the thing happen 30% of the time? This depends only on the values, and it is invisible to every metric in the column on the left.

That invisibility is exact, not approximate. Apply any strictly increasing function to every score — square it, take its log, run it through another sigmoid — and the ordering is identical, so AUC is identical, so NDCG is identical. Meanwhile every number has changed.

Break it on purpose

Below, 6,000 events with a known true probability and a real outcome. The model’s score is a monotone distortion of the truth, and the two sliders control the distortion. Because it is monotone, no setting can reorder anything.

A reliability diagram. Each point is a bin of events grouped by what the model predicted; horizontally the average prediction in that bin, vertically the fraction that actually happened. Perfect calibration is the dashed diagonal and nothing else. Calibrators are fitted on 3,000 events and every number is reported on the other 3,000.

Watch the AUC readout as you drag either slider. It sits at 0.8686 and does not move — not at any confidence, not at any bias. Now watch everything else.

At the default overconfident setting, the calibration error is 7.5% and the forecast is 10.8% low. Push the bias to +0.8 and the model over-forecasts by 32% while remaining exactly as good a ranker as it was. Pull confidence down to 0.4 with a bias of −1.5 and it under-forecasts by 57%.

Any of these would pass a ranking review.

Where the number stops being sorted

If your model’s output is only ever used to order a list, none of this matters and you can stop reading. That is rarer than it sounds. The number stops being a ranking and becomes a claim the moment it is:

Multiplied by anything Expected value is probability × value. Ad bidding, expected revenue ranking, anything that weighs several objectives together — all of them multiply a probability by a magnitude, and a probability that is 20% too high becomes a bid that is 20% too high.
Compared against a threshold “Auto-approve above 0.9”, “send for review below 0.3”, “hide if the abuse score exceeds 0.7”. A threshold is a statement about probability. Miscalibrate and the same threshold silently means something else.
Added up Forecasts, capacity planning, revenue projections, the number on the quarterly slide. Summing predictions is the single most sensitive use of a probability, because errors that would cancel in a ranking accumulate here.
Compared across models Two rankers, two heads of one model, an old version and a new one. Combining uncalibrated scores is exactly the problem that makes hybrid retrieval annoying, one level down.
Shown to a person “94% match”, “high risk”, “likely to sell out”. Once a number is on a screen you have made a promise about the world, and users calibrate themselves against it whether or not you did.

The two fixes

Both learn a map from the model’s score to a probability, and both must be fitted on data the model did not train on.

Platt scaling fits a logistic regression on the model’s log-odds:

Two parameters. It handles the two most common failure shapes — a model that is too sharp or too flat (), and one that is systematically high or low () — and it is stable on a few thousand events. Switch it on in the widget and the calibration error drops to about 2.4% at every slider position.

Isotonic regression fits the best non-decreasing step function through the outcomes, with no functional form at all. It can correct distortions no two-parameter family can. It also costs you something: in the widget it lands at 3.5% rather than Platt’s 2.4%, and it is the only method here that lowers AUC — from 0.8686 to 0.8640 — because collapsing a range of scores into one step creates ties that were not there before.

Platt when you have thousands of labelled events, or when the miscalibration looks like a smooth stretch or shift. Isotonic when you have tens of thousands and the reliability curve has a shape — an S-bend, a flat region, a model that is fine in the middle and wrong at both ends. If you are unsure, fit both and compare on a third split; it costs an hour.

Why models arrive miscalibrated

This is not a rare pathology. It is the default, and usually for one of four ordinary reasons.

Negative downsampling. Most conversion problems are 1% positive, so everyone throws away most of the negatives to make training tractable. The model then learns the odds of the sampled world, not the real one. This one has an exact correction, and it should be applied explicitly rather than left for a calibrator to absorb:

where is the model’s output and is the fraction of negatives kept. At a raw prediction of 0.5 corresponds to a true probability of 0.1. Nothing about that error is subtle, and it survives in production surprisingly often.

Regularization. L2 penalties, dropout and early stopping all shrink predictions toward the middle. The stronger the regularization, the flatter the model — which is a confidence below 1 on the widget’s first slider.

Class weighting. Up-weighting the positive class to “help with imbalance” does the same thing as downsampling and has the same effect on the odds. If both are in your pipeline, they compound.

Distribution shift. The base rate moves — seasonally, after a marketing push, after a change to who reaches the model at all. Calibration is fitted at one base rate and is wrong at another, which is why a calibrator that was correct in March is not automatically correct in September.

Practical rules

Fit on held-out data. Fitting a calibrator on the training set produces a curve that says the model is already calibrated, because on its own training data it very nearly is. This is the single most common implementation mistake and it fails silently — you ship a calibration layer that does nothing.

Calibrate per segment when the segments behave differently. A global calibrator applied to a model that is well calibrated on desktop and badly calibrated on mobile will split the difference and be wrong for both. Segment by whatever your base rate actually varies on — surface, country, new versus returning — but only where you have enough events per segment to fit anything.

Monitor it as a metric, not a launch checklist item. Expected calibration error and the ratio of predicted to actual over the last day are two numbers on a dashboard. When they drift, refit. Do not wait for someone in finance to find it.

Report both. A model release note that says “AUC 0.871, ECE 1.9%, forecast within 2%” is a complete claim. One that says “AUC 0.871” is half of one, and it is the half that nobody outside the team can spend.