What are you actually ranking for?

Ask a ranking team what they optimise and you will get one answer: click-through rate, or watch time, or conversion. Ask the same company’s finance team and you will get a different one. Ask the trust and safety team and you get a third.

All three objectives are already in your system. Only one of them is in the loss function. The other two are being handled by whoever complains loudest.

The mechanics: scalarization

The standard approach has two steps, and the first is the one people skip.

Normalize. A click probability lives in . A margin is measured in currency and might run from 0.4 to 10. Adding them means the margin wins every time, for the same reason that adding two retrievers’ scores hands the ranking to whichever produces bigger numbers. Map each objective onto a common range first — against the catalogue’s own spread, or against a fixed reference range if you want stability across days.

Weight and sum.

That is the whole method. It is used almost everywhere, it works, and its weakness is not accuracy — it is that is meaningless on its own. Nobody in the room can tell you whether 0.3 on margin is a lot.

Move the weights, watch the outcome

Sixty items, each with a click probability, an expected dwell time and a margin — and the tension every marketplace has built in: the things people click most are not the things that pay most. The top ten are served through a position-bias model, and the resulting clicks, seconds and revenue per session are computed exactly.

The knee is real and it is worth money

Start at the default and drag the margin slider up until the weights read roughly 50 / 10 / 40.

Clicks per session fall from 0.978 to 0.903 — down 8%. Revenue per session rises from 3.08 to 4.23 — up 37%. Dwell time barely moves.

That is not a clever trick, it is the shape of the frontier. Near a single-objective optimum the gradient of that objective is flat by definition: you are at a maximum. Everything else still has slope. So the first few percent of clicks you are willing to give up are the cheapest percent you will ever sell, and a system tuned to pure click-through is standing exactly on the spot where that trade is most favourable and refusing to make it.

Go gentler still: weights of 70 / 0 / 30 give up 2% of clicks for 23% more revenue.

The trap: weighting an objective is not maximising it

Now drag the sliders to pure margin — 0 / 0 / 100.

Revenue per session collapses to 1.42, a third of what the balanced weighting achieved and worse than ranking purely by click-through. Ranking by margin puts the most expensive items on the page, and nobody clicks them, and an item that is not clicked earns nothing.

Revenue is a product. The weight on margin is a term inside a ranking score, which is a completely different object. This confusion is common enough to have a signature: a business stakeholder asks for more weight on revenue, gets it, and revenue falls — after which everyone concludes the model is broken.

The general statement: the objective you weight is not the objective you get. Weight the components; measure the outcomes; never assume the two move together.

What scalarization cannot do

• It cannot reach a concave stretch of the frontier A linear score always lands on the convex hull of what is achievable. If the true frontier dips inward somewhere, no weighting reaches that region — the solution jumps past it as you turn the dial. So a target that seems impossible to hit may genuinely be unreachable by this method rather than badly tuned, and the fix is a constraint or a lexicographic rule rather than another afternoon of sliders.
• It cannot survive a model change unattended The weights multiply scores, so they are attached to a score distribution. Ship a model whose click head is sharper and the same 0.3 on margin is now effectively 0.2. Nothing errors. The trade you agreed to in a meeting quietly moves. This is the strongest practical argument for reporting outcomes rather than scores and for calibrating each head before combining.
• It cannot tell you the objectives are correlated Dwell time in this catalogue rises with click probability, so weighting it barely changes the list. Real objectives are often like this — you can spend a lot of effort tuning a knob that does nothing because it points in a direction another knob already covers. Sweep it and look before you argue about it.

The framing I would use instead

Rather than shipping weights, ship a constraint:

Maximise clicks per session, subject to revenue per session ≥ 4.0.

Then solve for the weights that satisfy it. Three things get better immediately. The commitment is stated in business units, so it survives a model change — you re-solve for new weights and the promise still holds. It is monitorable: revenue per session becomes a guardrail with an alert on it, rather than a hope. And it makes the failure explicit — if no weighting reaches 4.0, you learn that the target is infeasible with this catalogue, which is real information rather than a tuning failure.

Practically, that means keeping the sweep. The frontier is cheap to recompute — 231 weightings over 60 items here, and offline over a query sample it is a batch job — and it should be regenerated with every model release. Comparing this release’s frontier against last release’s is also the cleanest measure of whether the new model is actually better: a model that dominates the old frontier everywhere is an improvement, and one that merely moves along it just retuned the trade.

A last one, worth saying plainly

The objectives in the widget are known. In production every one of them is a prediction from a model, with its own error. A click head that is well calibrated and a margin estimate that is stale do not combine into a trustworthy trade — the weighting inherits the errors of both, and it inherits them multiplied by whatever weight you assigned.

Before spending a quarter tuning weights, check that each objective’s estimate is worth combining at all. That is a smaller and much more tractable problem, and it is usually where the improvement actually was.