{
  "api_version": "v1",
  "dataset_version": "AdityaNet_v2_dataset_r1",
  "dataset_sha256": "43fd0e228b28ae6bc7e468c3acf68722768bd62b73798eb6631e9e6233b71ed9",
  "generated_at": "2026-07-29T13:34:16.822019+00:00",
  "source_commit": "99af630",
  "data": {
    "title": "Model Comparison \u2014 Milestone XI",
    "date": "2026-07-18",
    "lead": "Performance, uncertainty, strengths, weaknesses, and statistical significance for every model, both tasks.\n\n---",
    "sections": [
      {
        "heading": "1. Nowcast \u2014 head-to-head",
        "body": "| Model | Strength | Weakness | Verdict vs threshold |\n|---|---|---|---|\n| Persistence | Perfect steady-state tracking (recall 1.0, 0 false) | **No onset detection** \u2014 needs the flare already flagged | Dominates, but operationally hollow (\u00a74) |\n| **Threshold (rate)** | **Fewest false alarms (15); simplest; free; interpretable** | Slightly lower recall (0.927) and AUC (0.954) | **Baseline \u2014 the one to beat** |\n| Logistic | Cheap; interpretable coefficients | Worst false-alarm count (228); poorly calibrated (Brier 0.029) | AUC +0.010 but **15\u00d7 more false runs** |\n| Random Forest | Highest AUC (0.966); best learned precision (0.766) | 4\u00d7 the threshold's false runs (61) | AUC +0.012, recall indistinguishable |\n| LightGBM | Highest event recall (0.988); calibrated (Brier 0.016) | 5\u00d7 the threshold's false runs (79) | AUC +0.007, more false alarms |\n\n**Statistical significance.** LightGBM \u2212 Threshold ROC-AUC = **+0.0076 [+0.0032, +0.0123]** \u2014 the CI excludes zero, so the difference is *real*, but it is **under one AUC point**. Event-recall CIs **overlap heavily** (threshold [0.875, 0.976] vs LightGBM [0.960, 1.00]), so on the operational metric the models are **statistically indistinguishable** from the threshold. The learned models' only consistent, significant edge is a fraction of an AUC point \u2014 bought with 4\u201315\u00d7 more false alarms."
      },
      {
        "heading": "2. Prediction \u2014 head-to-head",
        "body": "| Model | ROC-AUC [95% CI] | Event recall [95% CI] | Verdict |\n|---|---|---|---|\n| **Threshold (rate)** | **0.792** [0.708, 0.856] | 0.439 [0.250, 0.571] | **Best non-trivial model** |\n| Logistic | 0.780 [0.674, 0.850] | 0.463 [0.268, 0.600] | Indistinguishable from threshold |\n| Random Forest | 0.784 [0.694, 0.850] | 0.415 [0.220, 0.547] | Indistinguishable from threshold |\n| LightGBM | 0.768 [0.680, 0.831] | 0.439 [0.256, 0.565] | **Significantly worse** (paired \u0394 \u22120.023 [\u22120.045, \u22120.0004]) |\n\n**Every learned model's CI overlaps the threshold's on both metrics**, and the one significant paired result is LightGBM being *worse*. There is **no evidence that any learned model improves 30-min prediction over a threshold on the current rate.**"
      },
      {
        "heading": "3. What each model learned (interpretability)",
        "body": "**LightGBM feature importance (nowcast, top 8):** `roll_std_60` 0.141, `roll_mean_60` 0.127, `bg_excess` 0.108, `roll_max_15` 0.086, `rise_15` 0.086, `roll_std_15` 0.084, `log_rate` 0.081, `roll_mean_30` 0.080.\n\n**Logistic coefficients (nowcast, top |\u03b2|):** `log_rate` **+5.90**, `roll_std_15` \u22124.70, `roll_mean_15` \u22123.24, `gti_fraction` +2.30, `roll_max_15` +2.20.\n\n**Interpretation.** Both models concentrate on the **level and recent variability of the count rate**. The dominant logistic coefficient is `log_rate` \u2014 literally the quantity the threshold detector thresholds. The tree's top features (`roll_mean_60`, `roll_std_60`, `bg_excess`) are the **background activity level** identified in Milestone X. The models are not discovering a new signal; they are re-expressing \"the current and recent rate is elevated,\" which the threshold captures in one dimension. This is *why* the threshold is competitive: the learned models' feature space collapses onto the rate."
      },
      {
        "heading": "4. On persistence \u2014 why the strongest number is the least useful",
        "body": "Persistence achieves AUC 0.982 / recall 1.000 / 0 false runs by predicting `y(t) \u2248 y(t\u22121)`. For **nowcast**, this is legitimate (whether a flare was in progress a minute ago is known at time t) but tells you nothing until the flare has *already started* \u2014 it cannot detect onset, which is the operationally valuable moment. For **prediction**, persistence exploits that the \"flare within 30 min\" label barely changes minute to minute; it provides no lead time beyond the label's own smoothness.\n\n**Consequence for evaluation.** Steady-state recall is the wrong headline metric for nowcast \u2014 persistence maxes it trivially. The right metric is **detection latency at onset**, which persistence cannot address and which the threshold and learned models can. This should be the focus of any future nowcast work."
      },
      {
        "heading": "5. Calibration",
        "body": "Brier scores (lower better): Persistence 0.001, LightGBM 0.016, Random Forest 0.019, Logistic 0.029. The threshold detector is a hard classifier (no calibrated probability). Among learned models, **LightGBM is best calibrated**; logistic is notably worse and should not be used for probability outputs without recalibration."
      },
      {
        "heading": "6. Overall recommendation",
        "body": "**Nowcast:** the **threshold detector** is the recommended operational model \u2014 92.7 % event recall at 15 false runs, free, interpretable, and statistically indistinguishable from the learned models on event recall. If a calibrated probability is required, **LightGBM** is the learned alternative, at the cost of ~5\u00d7 more false alarms for < 1 AUC point.\n\n**Prediction:** the **threshold detector**, because no learned model beats it and one is significantly worse. But note the task ceiling: ~44 % event recall, dominated by activity-state persistence."
      }
    ],
    "source": "artifacts/v2/ml/MODEL_COMPARISON.md"
  }
}