Rare events are a measurement problem, not a modelling problem

Here is a claim I will defend: for most rare-event problems, the choice of model is nearly irrelevant, and almost all the variance in reported results comes from decisions about measurement.

Not the architecture. Not the hyperparameters. The split, the baseline, the metric, and the number of times you repeated the experiment.

The arithmetic that drives everything

In BEACON, high-risk conjunctions are 0.61% of events. A test split contains about twelve of them.

Twelve. Every metric you compute about the positive class is a statistic over twelve objects. Precision at the top 5% is a statement about how many of twelve landed in a bucket. Move one event and the number moves by eight percentage points.

Now consider what is usually done with a setup like that: fit several model families, compare their test scores, and report the winner. The gap between two of those scores is routinely smaller than the movement you would get by reshuffling which twelve events landed in test.

That comparison is not measuring the models. It is measuring the seed.

What repetition reveals

BEACON repeats the entire event-level split across 20 seeds and reports mean and standard deviation for every metric. The headline number at the one-day horizon is 0.806 PR-AUC with a standard deviation of 0.091.

That spread is the finding. It says a single run of this experiment could plausibly have produced 0.72 or 0.89. Anyone reporting 0.806 from one split, to three decimal places, is communicating a precision the data cannot support — and would have reported 0.72 with equal confidence had the shuffle gone differently.

The comparison against the baseline survives this. Current-risk ranking sits at 0.581 with a standard deviation of 0.085, and the learned model beats it at every horizon across all 20 splits. That is a claim the measurement supports. “Model A beats model B by 0.02” is not.

Three ways the measurement lies before the model gets a chance

The split leaks. Conjunction events generate multiple messages. Split by row and near-duplicates of one encounter appear in both train and test, so the model is rewarded for recognising events it has already seen rather than learning anything transferable. Splitting by event is one line and it changes the result.

The metric flatters. Accuracy on a 0.61% positive rate is meaningless — predict nothing and score 99.4%. ROC-AUC is better but still generous, because the enormous negative class keeps the false-positive rate small no matter what you do. Precision-recall is the honest lens when the positives are what you care about.

The baseline is a strawman. This is the one I would fix first in most published work. If your comparator is a random ranker, or logistic regression with default settings on unscaled features, you have not learned whether your method is good. You have learned that it beats something nobody would deploy.

BEACON’s baseline is the risk estimate that already ships in the data — a domain-computed quantity that operators already use. It is a hard comparator. It also wins on uncertainty-based escalation, where sorting by existing risk edges out sorting by ensemble disagreement at every horizon.

That result would have been trivially avoidable by choosing a weaker baseline. Keeping it is what makes the rest of the numbers worth reading.

The objection

The obvious reply is that models clearly do matter — gradient boosting beat logistic regression here, and that gap was consistent.

Agreed, and the qualifier is doing the work. The family mattered: a model that can express feature interactions beat one that cannot, by a margin that survived 20 reshuffles. What did not matter was anything finer. Distinguishing between two well-tuned members of the same family, at twelve positives per split, is beyond what this measurement can resolve.

So the practical rule is not “models are irrelevant.” It is: spend effort on model choice until you have crossed a family boundary, then stop and spend it on measurement instead. The marginal return on the second is much higher and almost nobody collects it.

What I would ask of any rare-event result

• What is the unit of independence, and does the split respect it?
• How many positives are in test? Say the integer.
• How many times was the split repeated, and what is the spread?
• What is the strongest baseline a domain expert would propose, and how does it do?
• Is the reported metric one the rare class can actually influence?

None of these are about the model. All of them determine whether the number means anything. A result that answers them with a modest score is worth more than a spectacular one that does not, and the difference is usually invisible from the abstract.