Skip to content

ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA

Findings · method

The full method

Every benchmark table, the adjudicated verdicts verbatim, and the protocol that was frozen before any model was fit.

← Back to the finding

What is this?

The full method behind the finding: benchmark tables for every task, the adjudicated verdicts verbatim, and the evaluation protocol.

Why should I trust it?

The protocol was frozen before any model was fitted, and the tables are rendered from the committed benchmark artifact rather than transcribed.

Where can I verify it?

Central question: does machine learning provide measurable operational value beyond strong classical baselines, for M/X flare nowcast and 30-minute prediction on AdityaNet_v2_dataset_r1?

Answer, stated plainly: No — not for these tasks on this dataset. A simple threshold on the SoLEXS count rate is operationally the strongest non-trivial detector, and the gradient-boosted and forest models do not beat it in any way that matters. This is a negative result, and per the milestone's own success criteria it is a successful outcome: an honest, reproducible, statistically defensible benchmark that future work can build on.

Each conclusion below follows the required structure.



Benchmark

M/X nowcast

train 297,295 · val 74,324 · test 192,541 minutes · test positives 1.24%

M/X nowcast: ROC-AUC and event recall with 95% confidence intervals, and false event runs, for eight models.
ModelROC-AUC · 95% CIEvent recall · 95% CIFalse runs
Randomtrivial
0.497[0.483, 0.509]
0.988[0.978, 1.000]
47,595
Majoritytrivial
0.500[0.500, 0.500]
0.000[0.000, 0.000]
0
Climatologytrivial
0.500[0.500, 0.500]
0.000[0.000, 0.000]
0
Persistencetrivial
0.982[0.978, 0.986]
1.000[1.000, 1.000]
0
Threshold (rate)
0.954[0.940, 0.966]
0.927[0.875, 0.976]
15
Logistic regression
0.964[0.953, 0.974]
0.976[0.929, 1.000]
228
Random forest
0.966[0.956, 0.976]
0.976[0.930, 1.000]
61
LightGBM
0.961[0.949, 0.972]
0.988[0.960, 1.000]
79

artifacts/v2/ml/benchmark_results.json


Benchmark

M/X 30-minute prediction

train 297,295 · val 74,324 · test 192,541 minutes · test positives 1.29%

M/X 30-minute prediction: ROC-AUC and event recall with 95% confidence intervals, and false event runs, for eight models.
ModelROC-AUC · 95% CIEvent recall · 95% CIFalse runs
Randomtrivial
0.493[0.483, 0.507]
1.000[1.000, 1.000]
47,578
Majoritytrivial
0.500[0.500, 0.500]
0.000[0.000, 0.000]
0
Climatologytrivial
0.500[0.500, 0.500]
0.000[0.000, 0.000]
0
Persistencetrivial
0.983[0.982, 0.984]
1.000[0.976, 1.000]
0
Threshold (rate)
0.792[0.708, 0.856]
0.439[0.250, 0.571]
208
Logistic regression
0.780[0.674, 0.850]
0.463[0.268, 0.600]
221
Random forest
0.784[0.694, 0.850]
0.415[0.220, 0.547]
243
LightGBM
0.768[0.680, 0.831]
0.439[0.256, 0.565]
473

artifacts/v2/ml/benchmark_results.json


Adjudication

Overall verdict

Rendered verbatim from the comparison report. These are adjudications made under a protocol frozen before any model was fit, not classifications this page computes.

Nowcast: the threshold detector is the recommended operational model — 92.7 % event recall at 15 false runs, free, interpretable, and statistically indistinguishable from the learned models on event recall. If a calibrated probability is required, LightGBM is the learned alternative, at the cost of ~5× more false alarms for < 1 AUC point.

Prediction: the threshold detector, because no learned model beats it and one is significantly worse. But note the task ceiling: ~44 % event recall, dominated by activity-state persistence.


Null result

Spectral resolution: a confirmed null

T1 only
0.9605
+ spectral bands
0.9638
Δ ROC-AUC
+0.0033

Ablation on the SoLEXS-only nowcast (371,619 train / 192,541 test minutes), LightGBM, base = {log_rate, roll_mean_15, roll_std_15, gti_fraction}:

Feature set ROC-AUC Δ vs T1-only
T1-only 0.9605
T1 + spectral bands (soft/mid/hard) 0.9638 +0.0033
T1 + hardness ratio 0.9618 +0.0013
T1 + all spectral 0.9634 +0.0029

Verdict: T2 spectral features add ≤ 0.003 AUC — a confirmed NULL. This was predicted in Milestone X from univariate analysis (≤ 0.012 AUC; hardness ratio at random for prediction). The multivariate LightGBM result is even smaller (+0.003), and well within the day-block-bootstrap CI width (~±0.012) of the base model. The 340-channel spectrum, summarised into bands, does not improve M/X detection at 1-minute resolution.

Scientific weight of this null. This is the central empirical test of v2's founding premise — that spectral resolution (340 real channels vs v1's 9 synthetic ones) would unlock performance. The evidence does not support that premise for flare detection. The result is reported as a contribution, not hidden: spectral resolution is scientifically valuable for characterising flares, but it does not measurably improve detecting or predicting them here.

Recommendation: EXCLUDE T2 from the operational feature set. Retain it only for future spectral-characterisation work (e.g. severity ranking once an RMF exists), not for detection.

artifacts/v2/ml/ablation_results.json


Method

Evaluation protocol

Frozen before any model was fit. Every choice below is justified by a measurement on the dataset, not by convention.

seed 20260718 · test from 2026-01-01 00:00:00+00:00 · 581 M/X events; day-block bootstrap CIs

1. Train / validation / test split

Choice. Chronological. Train 2024-02-01 → 2025-12-31; Validation = last 20 % of the pre-test period by time; Test = 2026-01-01 → 2026-06-15. Test opened once.

Evidence. Autocorrelation of log1p(rate_total): 0.997 @ 1 min, 0.804 @ 60 min, 0.643 @ 480 min. A random or k-fold split over minutes would place near-identical neighbours in both train and test, leaking the answer. The decorrelation time exceeds 8 hours, so only a temporal split with a gap at day granularity avoids leakage.

Why this boundary. M/X events by quarter: 2024Q1 96, 2024Q2 181, 2024Q4 6, 2025Q1 7, 2025Q2 3, 2025Q3 46, 2025Q4 87, 2026Q1 94, 2026Q2 61. The 2026-01-01 cut yields ~423 training events and ~155 test events — a substantial, contiguous, unseen test period spanning two quarters of genuine activity. Earlier cuts would test on the sparse 2025 tail (3–7 events/quarter); later cuts would leave too little test data.

2. Temporal split strategy

Choice. Single forward-chaining split (train → val → test in time), no shuffling, no random folds.

Evidence. Same autocorrelation result. Additionally, flare occurrence is non-stationary across the solar cycle; a forward split is the only one that measures generalisation to a future period, which is the operational question. Cross-validation over time would either leak (random folds) or waste the scarce events (blocked k-fold on 581 events leaves too few per fold for stable estimates).

3. Event grouping

Choice. Contiguous runs of positive-label minutes = one event. Metrics are computed at both the minute and the event level; the event level is primary.

Evidence. The dataset contains 564,160 usable minutes but only 581 independent M/X events (Milestone X). Treating minutes as independent samples overstates the effective sample size by ~970×. An event is the unit at which the label is actually independent.

Event-level definitions.

  • Event recall = fraction of true events with ≥ 1 predicted-positive minute.
  • False event runs = predicted-positive minute-runs not overlapping any true event (the operational false-alarm unit).

4. Handling of autocorrelation

Choice. Confidence intervals by day-block bootstrap: resample whole UTC days with replacement (1,000 replicates), recompute event recall and minute ROC-AUC on each.

Evidence. With minute-level autocorrelation of 0.64 even at 8 h, an IID bootstrap over minutes would produce absurdly narrow intervals. The day (1,440 min) is the practical independent block — it exceeds the strong-correlation window and matches the natural observing unit. This is the same moving-block principle the frozen v1 harness uses, adapted to day granularity.

5. Confidence intervals

Choice. 95 % CIs from the day-block bootstrap percentiles (2.5 / 97.5) for every reported metric on every model.

Evidence. 155 test events is small; point estimates without intervals would invite over-reading. The block bootstrap over days is the honest uncertainty given the autocorrelation structure measured in §4.

6. Statistical significance testing

Choice. For the headline comparison (best model vs threshold detector), a paired day-block bootstrap of the metric difference: resample days, compute Δ(event recall) and Δ(ROC-AUC) per replicate, and report the CI of the difference. If that CI contains zero, the two are declared statistically indistinguishable.

Evidence. Paired testing controls for day-to-day difficulty variation (some test days have large flares, some none). An unpaired comparison would be dominated by which days each bootstrap sample happened to draw. The paired difference isolates the model contribution.

artifacts/v2/ml/EVALUATION_PROTOCOL.md