Your split is leaking and your metrics are lying

A model scored well. The metrics were strong across the board. Then I changed one line in how the data was split and a chunk of the performance disappeared.

The line was which column the split key came from.

The shape of the problem

In BEACON, the raw records are conjunction data messages. A single close approach between two objects generates several of them over the days leading up to closest approach, as tracking updates and the risk estimate is revised.

So the data has a hierarchy. Many rows, fewer events. Rows belonging to the same event describe the same physical encounter, share most of their features, and carry the same label.

Split those rows randomly into train and test, and near-duplicates of the same encounter land on both sides. The model does not have to learn what makes a conjunction dangerous. It can learn to recognize this particular conjunction, which it has already seen, and it will be rewarded for that at test time.

The fix is to split by event_id, so every row from a given encounter stays together on one side of the boundary. It is a one-line change. It is also the difference between a result and an artefact.

This is not a niche concern

Grouped data is the normal case, not the exception:

• Multiple lab results from the same patient.
• Multiple sessions from the same user.
• Multiple photographs of the same physical site.
• Multiple readings from the same sensor or serial number.
• Multiple frames from the same video.

In every one of those, a random row-level split leaks identity across the boundary and inflates your metrics. The model looks like it generalizes to new patients when it has only generalized to new rows about patients it already knows.

The tell is a model that performs suspiciously well and then degrades in deployment on genuinely new subjects. By then the split has usually been forgotten.

Label leakage is the same mistake wearing a different hat

BEACON labels an event high-risk based on its final pre-closest-approach risk value. That label metadata — the final risk, and the time it was recorded — obviously cannot be a model input. It is the answer.

Excluding it is easy once you have said it out loud. The reason it needs saying is that these columns tend to live in the same dataframe as the features, one drop away from being included by accident, and a model trained with the answer in its inputs produces beautiful numbers.

Both failures share a root cause: information that will not be available at prediction time was available at training time.

One split is not a result

Fixing the split exposes a second problem. Once rows are grouped by event, the effective sample size collapses to the number of events, and in BEACON high-risk events are about 0.61% of the data — roughly twelve per test horizon.

Twelve. Which particular twelve land in your test set moves every headline number materially. A single split does not measure the method; it measures the seed.

The remedy is to stop trusting any single split: repeat it across many seeds and report the spread beside the mean. BEACON uses 20. Quite how much that reframes a result is a longer argument than belongs here, and one I make separately.

A short checklist

Before believing any evaluation number, including my own:

• What is the unit of independence in this data? Is the split key that unit?
• Could any feature encode the identity of a group that appears in both splits?
• Is any feature derived from information that only exists after the outcome?
• How many positives are in the test set? Would a different seed change the story?
• Does the reported number come with a spread, or is it a single point?

None of this makes a model better. It makes the measurement honest, which is a prerequisite for knowing whether the model is better at all.