Where early prediction fails

Every number in BEACON gets worse the earlier you try to predict. At one day before closest approach the model reaches 0.806 PR-AUC. At the earliest available observation it manages 0.233.

That is the result I would most like to have improved, and did not. Here is what the failure looks like from the inside.

The shape of the decay

Horizon	PR-AUC	Top-5% recall	10% escalation
1 day	0.806 ± 0.091	97.1%	97.5%
2 days	0.630 ± 0.106	94.6%	96.3%
3 days	0.493 ± 0.090	95.0%	97.5%
early	0.233 ± 0.082	71.3%	80.8%

Ranking degrades smoothly from one day to three, then falls off a cliff at the early horizon. Top-5% recall holds above 94% across the first three and drops 23 points at the last. Whatever happens between three days and “earliest available” is qualitatively different from the gentle decay before it.

The first honest thing: “early” is not a horizon

The study was designed around a 7-day horizon, and the data would not support one — a decision I mentioned in passing when writing about metrics and should explain properly here.

What I report instead is an early horizon. Its definition is the thing that matters here: whichever conjunction message happens to be the first one on record for that event. That definition has a property the others do not: it is not a fixed lead time. One event’s early might be six days out; another’s might be four. The column mixes lead times together.

So part of the collapse is measurement, not modelling. The 1-day, 2-day and 3-day columns each hold observations at a consistent distance from the event. The early column holds a distribution of distances, and averaging across it blurs whatever signal exists at any particular one.

I could have hidden this by calling it 7-day. The numbers would have looked the same and the label would have been false.

The second honest thing: the information may not be there

The more likely explanation is simpler and less satisfying. Conjunction risk estimates are refined as tracking accumulates. An early message describes an encounter whose geometry is still poorly constrained — the uncertainty on the miss distance is large, and the eventual outcome depends on updates that have not happened yet.

The baseline supports this reading. Ranking directly by the current risk estimate scores 0.109 at the early horizon against 0.581 at one day. The domain signal degrades just as sharply as the learned one. If the feature everyone agrees is informative loses most of its power, it is unlikely that a better model recovers it from the remainder.

The learned model still roughly doubles the baseline, 0.233 against 0.109. That ratio is preserved across every horizon. What changes is the absolute level, and the level is set by how much the data knows.

What the escalation numbers still permit

The one column that degrades least is uncertainty-based escalation: 80.8% of high-risk events captured by reviewing the 10% most uncertain, against 8.3% for random selection. Worse than the 97.5% at one day, and still nearly ten times better than chance.

That suggests a use for the early horizon that ranking does not support. The model cannot reliably tell you which early events are dangerous. It can tell you which ones it cannot yet judge — and at a stage where the honest answer is “wait for more tracking data,” a system that flags its own ignorance is doing the useful thing.

Which reframes the negative result rather than rescuing it. Early prediction fails as prediction. It survives as triage of attention.

What I have not ruled out

Being clear about the limits of a negative result:

• Sample size. About twelve high-risk events per test split. Across 20 repeated splits the early-horizon standard deviation is ±0.082 on a mean of 0.233 — a third of the value. That is a wide interval to draw conclusions inside.
• Features. I used the fields present in the messages. Orbital elements, object characteristics or sensor metadata might carry early signal the messages do not expose.
• Sequence structure. Every horizon is evaluated as an independent snapshot. A model over the trajectory of successive messages might extract something a single early observation cannot.
• The label. High risk is defined by final pre-encounter risk crossing a threshold. A different threshold changes the positive rate and could change the difficulty at every horizon.

Any of those could turn this around. None of them are ruled out by what I ran, and saying so is part of reporting the result.

Why publish it

The early horizon is the one an operator would most want, which is exactly why the negative result matters more than the positive ones. A study that reported only the 1-day number would imply a capability the data does not support.

The horizon construction, the coverage diagnostics and every number above are in the repository if you want to check the reasoning.

References

1. T. Saito, M. Rehmsmeier. “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets.” PLOS ONE, 10(3), 2015.
2. T. Uriot et al. “Spacecraft Collision Avoidance Challenge: Design and Results of a Machine Learning Competition.” Astrodynamics, 2022. The prior work establishing this dataset and framing.
3. D. Kendall, Y. Gal. “What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?” NeurIPS, 2017. Separating uncertainty that more data would reduce from uncertainty inherent to the problem.
4. D. J. Kessler, B. G. Cour-Palais. “Collision Frequency of Artificial Satellites: The Creation of a Debris Belt.” Journal of Geophysical Research, 83(A6), 1978. Why early warning is the horizon anyone actually wants.
5. A. Rogers. “How the Transformers Broke NLP Leaderboards” and related work on publication bias. Context for why negative results are undersupplied.