Accuracy is the wrong question for rare events

BEACON is a research prototype for satellite conjunction triage: deciding which close approaches between objects in orbit deserve a human analyst’s attention. The input is public conjunction data messages. The label is whether an event turns out to be high-risk.

In the test splits, 0.61% of events are high-risk. About twelve events per horizon, out of roughly two thousand.

So here is a model. It looks at any conjunction event and says “not high-risk.” Every time. No features, no training, four characters of Python.

def predict(event):
    return 0

That model is 99.4% accurate. It is also completely worthless. It would let every single dangerous conjunction through unflagged.

This is not a clever edge case. It is the normal condition for any rare-event problem — fraud, disease screening, equipment failure, collision risk. And it means that the first question people reach for, “how accurate is it?”, is not just insufficient. It actively misleads.

What to measure instead

If accuracy is out, what replaces it? The honest reframe is to stop asking whether the model classifies correctly and start asking the question an operator actually has: if I can only review a handful of events today, will the right ones be at the top?

That turns triage into a ranking problem, and ranking has its own metrics:

• PR-AUC instead of ROC-AUC. Precision-recall focuses on the rare positive class; ROC curves can look reassuring on imbalanced data because the huge negative class dominates the false-positive rate.
• Top-K recall. If I review the top 5% of ranked events, what fraction of the genuinely high-risk ones did I catch? This maps directly onto a real review budget.

Measured that way, the learned models earn their place. Across 20 repeated event-level splits, gradient boosting improved PR-AUC over the existing risk estimate at every warning horizon:

Horizon	Learned PR-AUC	Current-risk PR-AUC
1 day	0.806 ± 0.091	0.581 ± 0.085
2 days	0.630 ± 0.106	0.367 ± 0.083
3 days	0.493 ± 0.090	0.237 ± 0.048
early	0.233 ± 0.082	0.109 ± 0.031

At the top 5% review level, the ensemble captured about 97% of high-risk events one day out. That is a number an operator can reason about. “99.4% accurate” is not.

Ranking well and being right are different things

Ranking quality and probability quality are separate properties. A model can rank events almost perfectly and still produce probabilities that mean nothing.

Ranking only depends on order. If the model assigns 0.9 to the risky events and 0.4 to the safe ones, the ranking is perfect — but if the true rate among the 0.9 group is 3%, then “0.9” is a lie. Anyone who reads that number as a probability and makes a decision on it has been misled by a model that scores well on every ranking metric.

Calibration is the separate question of whether a predicted probability matches observed frequency. Among everything the model called 10%, does roughly 10% actually occur? I evaluated it with Brier score, Expected Calibration Error, and reliability curves, and fixed it with sigmoid calibration fitted on a validation split.

One practical wrinkle: in rare-event settings almost all predicted probabilities pile up near zero, so the usual evenly-spaced reliability bins are nearly unreadable — the leftmost bin swallows everything. Quantile binning, where each bin holds the same number of events rather than the same width of probability space, made the curves interpretable again.

Calibration barely moved the ranking, which is the expected result: it rescales probabilities without reordering them. Brier score and ECE both improved. That is the trade you want — same ordering, more honest numbers.

Uncertainty as its own signal

The more useful signal was not the prediction. It was the model’s disagreement with itself.

A bootstrap ensemble gives you this almost for free: train several gradient boosting models on resampled training data and record how far apart their predictions land. That spread is an uncertainty score — how much do plausible versions of this model disagree about this event?

High-risk events had substantially higher predictive uncertainty than routine ones. The model was not only assigning them higher risk — it was also less sure about them. That makes uncertainty usable as a triage signal in its own right. Escalating the 10% most uncertain events captured:

Horizon	By uncertainty	By current risk	Random
1 day	97.5%	99.6%	8.3%
2 days	96.3%	97.9%	9.6%
3 days	97.5%	97.9%	11.3%
early	80.8%	84.6%	8.3%

The column that matters most is the middle one

Look at that table again. Escalating by uncertainty beats random escalation by an enormous margin — 97.5% against 8.3%. It would be easy to stop there and write a much more exciting post.

But escalating by the risk estimate the data already ships with does slightly better, at every horizon. The domain signal that existed before I trained anything is a stronger comparator than the thing I built.

That result only exists because the baseline was chosen honestly. Comparing uncertainty escalation against random escalation alone would have produced a more impressive number and a less useful one. The defensible claim is narrower: uncertainty is a complementary review signal, not a replacement for domain risk.

The feature ablation points the same way. Strip the existing risk estimate out of the model’s inputs and PR-AUC drops at every horizon. The stripped model still ranks better than sorting by that risk estimate directly. So the other CDM features carry independent signal, and the existing risk value is central without being the whole story. Both halves of that sentence matter.

The plumbing that decides whether any of it is real

Three evaluation choices matter more here than the choice of model.

Split by event, not by row. Each conjunction event contributes several messages over time, so a random row-level split puts near-duplicates of the same encounter on both sides of the boundary. It is a one-line change and it moved the numbers enough that I gave it its own write-up.

Repeat the split. With about twelve positives per test set, a single split is close to meaningless — which particular dozen events land in test can swing every headline number. I ran 20 splits and report mean and standard deviation. Some of those standard deviations are large, and they should be visible.

Drop horizons the data cannot support. A planned 7-day horizon did not survive contact with the data, so I report an early horizon instead and define it as what it actually is: the earliest observation available per event. Those are the weakest numbers in the project and the ones I trust most.

What I would take to another problem

Very little of this is specific to orbital mechanics. If you are working on anything where the interesting outcome is rare:

• Accuracy is noise. Report PR-AUC and top-K recall against a real review budget.
• Ranking quality and probability quality are separate axes. Measure both; a model can pass one and fail the other.
• Compare against the strongest baseline you can construct, especially if it is a simple domain heuristic. If it beats you, that is a finding.
• Ensemble disagreement is cheap to compute and tells you where to send a human.
• Check what your split is actually leaking before you believe any metric.

BEACON is a research prototype, not an operational system, and I have tried to keep that line bright throughout. It runs on public data, the positive class is small even after repeated splits, and nothing in it has been validated in an operational environment.

The same problem shows up outside orbit. A property intelligence report has to distinguish a direct observation from a derived estimate from an analyst inference, or the reader cannot tell which findings are solid — which is the subject of a companion piece on the Arden Meridian blog. Different domain, same discipline: say what the number is, and say what it is not.

Everything above is reproducible: the code, the manuscript and the archived releases are on GitHub. If you work on rare-event evaluation and think I have got something wrong, tell me.