Findings · validation

The proof

Every figure below is computed from the 192,541 held-out predictions in the committed artifact — not summarised, not redrawn by hand.

← Back to the finding

What is this?

The scientific validation of the flare-detection result: ROC and precision-recall curves, calibration, confusion matrices, the threshold trade-off and a per-day error analysis.

Why should I trust it?

Every curve is computed at build time by a committed script from the 192,541 held-out predictions in the benchmark artifact. The scalar metrics are read from the published benchmark rather than recomputed, so the figures cannot disagree with the tables.

Where can I verify it?

Trace any figure hereClaim → artifact → commit
The model cardsFailure modes and limitations
The evaluation protocolFrozen before any model was fit
The curve geometry, as JSON/api/v1/findings/metrics.json

Ranking quality

ROC and precision–recall

ROC-AUC says the detectors are equivalent. Precision–recall says they are not. At a 1.24% base rate the PR curve is the honest one — ROC is dominated by the 190,148 negatives.

ROC

Threshold (count rate)AUC 0.954
LightGBMAUC 0.961

Nearly indistinguishable — this is the basis of the headline verdict.

Precision–recall

Threshold (count rate)AUC 0.485
LightGBMAUC 0.706

Here they separate: LightGBM ranks minutes substantially better. The dashed line is the base rate — where a random detector would sit.

Why the verdict is still the threshold

Better minute-level ranking did not become more detected flares. Event recall is statistically indistinguishable, and the learned model pays for its ranking with roughly five times the false-alarm runs. The trade-off section below shows that cost directly.

Every metric

Minute-level and event-level

Trivial baselines are shown because they bound the problem — and because Persistence beating everything is itself the finding.

Minute-level and event-level metrics for every detector.
Model	Precision	Recall	F1	MCC	Brier	PR-AUC	Event recall	False runs
Randomtrivial	0.012	0.501	0.024	-0.000	0.3338	0.012	0.988	47,595
Majoritytrivial	0.000	0.000	0.000	0.000	0.0124	0.012	0.000	0
Climatologytrivial	0.000	0.000	0.000	0.000	0.0127	0.012	0.000	0
Persistencetrivial	0.966	0.964	0.965	0.965	0.0009	0.932	1.000	0
Threshold (count rate)	0.548	0.589	0.568	0.562	—	0.486	0.927	15
Logistic regression	0.540	0.537	0.539	0.533	0.0294	0.556	0.976	228
Random forest	0.766	0.590	0.667	0.669	0.0191	0.699	0.988	47,595
LightGBM	0.745	0.628	0.681	0.680	0.0155	0.707	0.988	79

artifacts/v2/ml/benchmark_results.json

Confusion

What each detector actually did

Threshold (count rate)

True positive

1,409

False positive

1,163

False negative

984

True negative

188,985

LightGBM

True positive

1,503

False positive

515

False negative

890

True negative

189,633

Counted over 192,541 held-out minutes, of which 2,393 are flaring.

Calibration

The probabilities are not trustworthy

Reliability diagram

LightGBMBrier 0.0155

Perfect calibration lies on the diagonal. The curve sits far below it: when this model says 30%, the event occurs about 2% of the time.

Why this matters

The adjudicated verdict offers LightGBM as the option "if a calibrated probability is required". This diagram shows that as published, it is not calibrated — it is systematically over-confident. Anyone using these scores as probabilities would need to recalibrate first.

Reported here rather than omitted, because a limitation that only appears when someone deploys the model is a limitation the platform should have stated.

Trade-off

Recall costs alarms

Recall vs alarm burden

Threshold detector

Each point is a candidate threshold. Recall rises steeply at first, then buys very little for a rapidly growing alarm burden.

Operating point

Threshold: 6.233
Precision: 0.548
Recall: 0.589
F1: 0.568
Alarm runs: 90

Swept across 61 candidate thresholds in the log domain, because the count rate spans orders of magnitude. Runs, not raw false positives — an operator is interrupted once per run.

Error analysis

Where it fails

Errors are not spread evenly. Of 148 held-out days, 44 contain at least one error, and the misses concentrate in a handful of them.

Days with the most missed flaring minutes.
Date	Missed (FN)	False alarms (FP)	Caught (TP)
2026-03-28	127	0	73
2026-01-21	58	16	16
2026-04-24	57	58	85
2026-04-04	57	13	43
2026-02-01	52	53	102
2026-04-26	49	11	30

The dominant failure is a missed slow riser: a flare whose count rate climbs gradually and never crosses the threshold, producing many false negatives on a single day with no false positives. This is the specific weakness a learned model would need to fix to justify its alarm cost — and on event recall, it did not.

Ablation

Spectral resolution adds nothing

T1 (count rate) only

0.9605

+ spectral bands

0.9638

Δ ROC-AUC

+0.0033

Adding spectral bands moves ROC-AUC by 0.0033 — within noise. A confirmed null, reported because a null that goes unpublished is a result someone else will pay to rediscover.

artifacts/v2/ml/ablation_results.json

Provenance

How these figures were produced

01
Source
artifacts/v2/ml/benchmark_predictions.json — 192,541 held-out labels and per-sample scores.
02
Derivation
web/scripts/derive_metrics.py sweeps every distinct score to build ROC/PR, bins probabilities for calibration, and aggregates errors by day.
03
Cross-check
ROC-AUC computed from the curve reproduces the published ROC-AUC to five decimal places.
04
Render
Static SVG at build time — this page ships no JavaScript.

The full methodBenchmark tables, adjudicated verdicts, frozen protocol Reproduce itEnvironment, digests, rebuild record