Features that know how the story ends

A feature pipeline is a join. Somewhere in it is a line that computes, for each item, how often it has been clicked — and unless somebody deliberately stopped it, that line ran over the whole table.

Which means a training row from March carries a click rate that includes April, May and June. The model is being taught with a number that will not exist when it is asked to make the same prediction for real.

Point-in-time correctness, stated once

For every training row, every feature must be computed from data that was available strictly before the event the row describes.

That is the whole rule. It is easy to state, easy to agree with, and quietly violated by almost every pipeline that has not been designed to prevent it — because the natural way to build a feature table is one aggregate over everything, and the natural way to build a training set is to join it on.

Turn the leak up

Below, 400 items over 120 days. Item click rates drift — an item is hot for a fortnight and then is not — and the drift persists from day to day, which is what makes the future informative about the present. The model has two features: the item’s trailing click rate, and an honest content signal available at serve time.

The slider moves the right edge of the click-rate window forward from the day before the event into the future. Nothing about the world changes when you move it. Only how the training set was assembled changes.

At fourteen days of leakage, validation AUC reads 0.756. Production reads 0.730. Two and a half points of the offline result do not exist.

Notice the shape: the gap peaks around a fortnight and then falls again at thirty days. A very wide future window dilutes the informative near-future with distant days that say little about this one. So the size of the inflation is not even a monotone function of how badly you leaked, which rules out estimating it by eye.

The surprise: the model is not necessarily worse

Look at the fourth readout — the leaky model against a point-in-time model on the identical future traffic. It is slightly positive. Training with the leak produced a model that is, if anything, marginally better in production.

That is not a bug in the simulation, and it is worth understanding because it is why this class of error survives so long.

The leaky click rate is computed over more data than the honest one, so it is a less noisy estimate of the same underlying quantity. Training on a cleaner version of a feature teaches the model to weight that feature more, and at serving time the noisier version is still the best signal available — so the larger weight is roughly the right call anyway.

This connects directly to the offline–online gap: leakage does not merely add noise to your screening test, it adds a bias that varies by candidate. Nothing degrades a gate faster.

Where it actually comes from

• The whole-table aggregate SELECT item_id, SUM(clicks)/SUM(impressions) FROM events GROUP BY 1, joined onto training rows from any date. This is the one in the widget and it is by far the most common.
• The snapshot join A feature table that holds current values — price, rating, stock, follower count — joined onto historical events. Every row gets today’s value of a property that has changed since. No aggregate is involved, so it does not look like a leak.
• The centred window A “7-day rolling average” implemented as ±3 days. Time-series libraries default to centred windows, and a centred window always contains the event’s own day.
• Target encoding without out-of-fold Replacing a category with the mean label for that category, computed over the training set including the row itself. In a category with six rows, the feature is one sixth the label.
• Late-arriving data The subtlest one. A conversion attributed back to a click three days later is correct in the warehouse and unavailable at serve time. A feature that counts conversions “as of” a timestamp is only honest if it also respects when each conversion landed, not when it is dated.

Why a time-based split does not save you

The standard advice for temporal data is to validate on a later time period than you train on. Do that — it catches drift, and it is a much better check than a random split.

It does not catch this. If your feature pipeline computes aggregates over the whole table, it computes them over the whole table for the validation rows too. Both sides of the split are inflated, both by roughly the same mechanism, and the comparison looks clean. The leak is inside each row, not across the boundary between the two sets.

The only thing that catches it is computing the features correctly in the first place.

What actually fixes it

An as-of join, not a plain join. The primitive you need is: for each event at time , take the most recent feature value with a valid-from timestamp at or before . Every feature store is essentially a well-engineered version of this operation, and it is the reason feature stores exist. If you are not going to use one, write the as-of join by hand and put it behind a single function that every training set is built through.

Store features with two timestamps. When the value became true, and when your system learned it. Serving can only use rows whose learned-at is before the prediction. This is the only construction that handles late-arriving data correctly, and it is the difference between a feature table that is auditable and one that is merely usually right.

Log features at serve time. The strongest guarantee available: write the exact feature vector the model scored, at the moment it scored it, and train on that. It cannot leak, because it was produced by a system that had no access to the future. It costs storage and it fixes the whole class of problem at once — including training–serving skew from code that differs between the two paths, which is the same bug wearing a different hat.

Test for it. A pipeline test that builds one training row for a synthetic event, then adds a future event and rebuilds, and asserts that the feature vector is byte-identical. It takes an afternoon, it runs in CI, and it fails loudly the next time somebody adds a convenient aggregate.

The reason this is worth being strict about

Most data-quality problems announce themselves. Point-in-time violations do the opposite: the metrics get better, the model looks stronger, and everyone involved is pleased. There is no error, no alert, and no stage at which the pipeline behaves unexpectedly.

The only defence is that the number cannot be produced wrongly in the first place — which makes this an architecture decision rather than a code-review one, and one worth making before the first model rather than after the third disappointing launch.