ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA
Model cardslightgbm
Model card · candidate detector
LightGBM
Gradient-boosted decision trees.
What is this?
A model card for LightGBM, one of eight detectors evaluated on the frozen Aditya-L1 flare benchmark.
Why should I trust it?
Every figure below is read at build time from the committed benchmark artifact; none is transcribed. The judgements are traceable to the written evaluation protocol and the dataset limitations record.
Where can I verify it?
- The validation evidenceComputed from 192,541 held-out predictions
- Dataset limitationsL-1 … L-10
- Evidence traceabilityclaim → artifact → commit
- The raw payloadfindings/models.json
Section 1
Purpose and intended use
Why it is in the benchmark. The strongest candidate in the benchmark and the one that makes the negative result worth publishing: it is what a practitioner would actually reach for.
Intended use. Ranking minutes by flare likelihood, and testing whether better ranking translates into better operational detection. On this dataset it does not.
Out of scope. Its ranking advantage must not be reported as a detection advantage. At the base rate of this dataset, ranking and alarm burden come apart, and the operational question is settled by the second, not the first.
Section 2
Inputs, outputs, training
Inputs. Fourteen features per minute, all derived from the SoLEXS 1-minute count rate in T1: the log rate, rolling means over 5/15/30/60 minutes, a 15-minute rolling maximum, rolling standard deviations, a background-excess term, two rise rates, and three data-quality terms (GTI fraction, seconds present, partial-minute flag).
Outputs. A calibrated-scale probability per minute; a reliability diagram is published.
Training protocol. Chronological split with a held-out test period beginning at the frozen test start date; no shuffling, so no future minute can inform a past one. The protocol — split, seed, metric set and bootstrap scheme — was fixed in EVALUATION_PROTOCOL.md before any model was fitted.
Operating point. Decision threshold 0.9206, selected on the training period and then frozen.
Section 3
Evaluation
Both tasks, both reported.
M/X NOWCAST · 192,541 held-out minutes · base rate 1.24%
| Metric | Value | What it measures |
|---|---|---|
| ROC-AUC | 0.961 | Ranking quality across all thresholds |
| PR-AUC | 0.707 | The informative curve at a low base rate |
| Precision | 0.745 | Of the minutes flagged, how many were flares |
| Recall | 0.628 | Of the flare minutes, how many were flagged |
| F1 | 0.681 | Harmonic mean of the two above |
| MCC | 0.680 | Correlation, robust to imbalance |
| Brier | 0.016 | Probability accuracy; undefined for hard classifiers |
| Balanced accuracy | 0.813 | Mean of the two class recalls |
- Events detected
- 81 / 82
- Event recall 0.988 · 95% CI 0.960–1.000
- False alarm runs
- 79
- Of 171 alarm runs raised. A run, not a minute — the unit an operator actually responds to.
- ROC-AUC interval
- 0.949–0.972
- 95%, day-block bootstrap.
M/X 30-MIN PREDICTION · 192,541 held-out minutes · base rate 1.29%
| Metric | Value | What it measures |
|---|---|---|
| ROC-AUC | 0.768 | Ranking quality across all thresholds |
| PR-AUC | 0.103 | The informative curve at a low base rate |
| Precision | 0.179 | Of the minutes flagged, how many were flares |
| Recall | 0.235 | Of the flare minutes, how many were flagged |
| F1 | 0.203 | Harmonic mean of the two above |
| MCC | 0.193 | Correlation, robust to imbalance |
| Brier | 0.037 | Probability accuracy; undefined for hard classifiers |
| Balanced accuracy | 0.611 | Mean of the two class recalls |
- Events detected
- 36 / 82
- Event recall 0.439 · 95% CI 0.256–0.565
- False alarm runs
- 473
- Of 540 alarm runs raised. A run, not a minute — the unit an operator actually responds to.
- ROC-AUC interval
- 0.680–0.831
- 95%, day-block bootstrap.
Confusion matrix · M/X NOWCAST
True negative
189,633
False positive
515
False negative
890
True positive
1,503
Section 4
Feature attribution
Which of the fourteen features the fit relied on.
Attribution is not explanation. These features are strongly collinear — four of them are rolling means of the same series — so credit assigned to one is credit withheld from another, and none of it constitutes a physical account of flare onset.
Section 5
Strengths and failure modes
Strengths
- The best minute-level ranking of any model evaluated.
- Emits probabilities that can be checked against observed frequency.
Failure modes
- Raises substantially more false alarm runs than the threshold at comparable recall — the finding that decides the benchmark.
- Effective capacity outruns the event count, so apparent gains may not transfer to a different solar-cycle phase.
Section 6
Limitations and ethical considerations
Ethical considerations. This is the model a positive-result incentive would push to the front page. It is published with the result that does not flatter it, which is the point.
Governing limitation clauses
Clause text is not restated here. It is published in full on the dataset card, from the frozen DATASET_LIMITATIONS_FOR_ML.md report, so there is exactly one copy of every caveat.