ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA
Evidence · model cards
Every detector, documented
Eight models were evaluated on the same frozen split under a protocol fixed before any of them was fitted. Each has a card stating what it is for, what it must not be used for, and how it fails.
What is this?
A model card for each of the eight detectors in the flare benchmark — four candidate detectors and four trivial reference points.
Why should I trust it?
Every metric on these cards is read at build time from the committed benchmark artifact. The descriptive text is kept in a separate module that contains no numbers, so a figure cannot survive in prose after the artifact behind it changes.
Where can I verify it?
- The validation evidenceROC, PR, calibration, error analysis
- Evaluation protocolartifacts/v2/ml/EVALUATION_PROTOCOL.md
- Evidence traceabilityclaim → artifact → commit
- The benchmark payloadfindings/models.json
Shared protocol
What every card was measured under
The cards differ in what the model does. They do not differ in how it was judged.
- Seed
- 20260718
- Fixed before fitting.
- Test start
- 2026-01-01
- Chronological, never shuffled.
- Held-out minutes
- 192,541
- Training: 297,295.
- Base rate
- 1.24%
- Why PR-AUC, not accuracy.
Effective sample size. 581 M/X events; day-block bootstrap CIs. The row count is not the sample size — flares last many minutes, so the minutes are not independent. Confidence intervals are computed by bootstrapping whole days, which is the unit that can be treated as exchangeable.
The cards
Eight models
Candidate detector
Threshold on the count rate
Raises an alarm whenever the SoLEXS 1-minute count rate exceeds a single fixed value.
- ROC-AUC
- 0.954
- PR-AUC
- 0.486
- False runs
- 15
Read the card →
Candidate detector
LightGBM
Gradient-boosted decision trees.
- ROC-AUC
- 0.961
- PR-AUC
- 0.707
- False runs
- 79
Read the card →
Candidate detector
Random forest
An ensemble of decision trees, averaged.
- ROC-AUC
- 0.966
- PR-AUC
- 0.699
- False runs
- 61
Read the card →
Candidate detector
Logistic regression
A linear model over the fourteen features, squashed to a probability.
- ROC-AUC
- 0.964
- PR-AUC
- 0.556
- False runs
- 228
Read the card →
Reference point
Persistence
Predicts that the next label equals the previous one.
- ROC-AUC
- 0.982
- PR-AUC
- 0.932
- False runs
- 0
Read the card →
Reference point
Climatology
Predicts the historical base rate, ignoring the observation.
- ROC-AUC
- 0.500
- PR-AUC
- 0.012
- False runs
- 0
Read the card →
Reference point
Majority
Always predicts the majority class.
- ROC-AUC
- 0.500
- PR-AUC
- 0.012
- False runs
- 0
Read the card →
Reference point
Random
Assigns a uniform random score to every minute.
- ROC-AUC
- 0.497
- PR-AUC
- 0.012
- False runs
- 47,595
Read the card →
What the models used
Feature attribution
Three models, three different accounts of which of the fourteen features mattered — and none of them a physical explanation.
Attributions are shown on each model's own card, in their own units: signed coefficients for the linear model, unsigned importances for the two tree ensembles. They are not plotted on a shared axis, because they do not measure the same thing.