{
  "api_version": "v1",
  "dataset_version": "AdityaNet_v2_dataset_r1",
  "dataset_sha256": "43fd0e228b28ae6bc7e468c3acf68722768bd62b73798eb6631e9e6233b71ed9",
  "generated_at": "2026-07-29T13:34:16.825899+00:00",
  "source_commit": "99af630",
  "data": {
    "title": "Dataset Limitations for ML \u2014 `AdityaNet_v2_dataset_r1`",
    "date": "2026-07-18",
    "lead": "How the frozen dataset constrains **model choice**, **evaluation**, **loss functions**, **validation strategy**, and **achievable scientific claims**. Each limitation is stated with the measurement behind it and the concrete constraint it imposes.\n\n---",
    "sections": [
      {
        "heading": "L-1 \u2014 The effective sample size is 581, not 564,160",
        "body": "**Measurement.** 564,160 usable minutes contain **581 independent M/X events** (4,065 \u2265C; **47** X-class; **179** M/X on combined days).\n\n**Constrains model choice.** Model capacity must be sized to ~581 events, not to half a million rows. A model with thousands of effective parameters will fit event-specific noise. **Recommendation: \u2264 ~15 features and a low-capacity learner for the first benchmark**; deep sequence models only if Stage 3 evidence justifies them.\n\n**Constrains evaluation.** Minute-level metrics overstate confidence by roughly 564,160/581 \u2248 **970\u00d7**. **All metrics must be event-level**, with confidence intervals from a block bootstrap over events. Reporting minute-level AUC as the headline would be a statistical error, not a stylistic choice.\n\n**Constrains claims.** Any subgroup smaller than ~50 events cannot support a quantitative claim. This rules out standalone X-class conclusions (47 events; ~15\u201320 in a test split, where one event moves recall by ~5 points)."
      },
      {
        "heading": "L-2 \u2014 Forecast skill is horizon-flat: it is activity-state persistence, not precursor detection",
        "body": "**Measurement.** Prediction AUC on **quiet minutes only** (in-progress flares excluded): 30 min **0.8119**, 60 min 0.8023, 120 min 0.7951, 360 min **0.7884**. A **12\u00d7 horizon change costs 0.024 AUC**.\n\n**Constrains claims \u2014 severely.** Genuine precursor information must decay with horizon. Near-flat skill across 30 min to 6 h is the signature of a slowly varying activity state that persists for days. **A raw forecasting AUC of ~0.80 must never be reported as flare-prediction skill.** It must be reported as *improvement over a persistence baseline*, with the skill-vs-horizon curve published alongside.\n\n**Constrains validation.** Persistence and climatology baselines are **mandatory**, not optional \u2014 without them the number is uninterpretable. This is v1's forecast-vs-persistence finding reproduced on real data, now with the mechanism identified.\n\n**Constrains loss functions.** Optimising a horizon-agnostic loss will find the persistence solution, because that is where the gradient is. If precursor detection is the goal, the objective must explicitly reward *short*-horizon discrimination over long \u2014 otherwise the model will correctly learn \"the Sun is active\" and the result will be misread."
      },
      {
        "heading": "L-3 \u2014 No instrument response: physical severity targets are unconstructible",
        "body": "**Measurement.** No RMF/ARF anywhere in the archive. Channels are ordinal. The light curve is a band-limited integral of the spectrum with LC/\u03a3(PI) ratio varying **1.76\u201320.4** (F-1).\n\n**Constrains model choice.** No regression target in physical units (W/m\u00b2, GOES class as a continuous quantity) can be built or validated. **Calibrated severity regression is excluded** \u2014 this is a data limitation no architecture resolves.\n\n**Constrains claims.** Permitted: \"the model ranks events by observed intensity,\" \"the model classifies against GOES-defined labels.\" **Not permitted:** \"the model estimates GOES flux,\" \"the model measures plasma temperature,\" or any statement in keV.\n\n**Constrains features.** No physically-calibrated spectral feature (temperature, emission measure, spectral index) is derivable. Spectral features are ordinal-channel statistics only."
      },
      {
        "heading": "L-4 \u2014 The combined-instrument arm is 171 days",
        "body": "**Measurement.** SoLEXS-only: **424 days, 581 M/X events**. Combined SoLEXS+HEL1OS: **171 days (2025-12-07 \u2192 2026-06-15), 179 M/X events, 13 X-class**.\n\n**Constrains validation.** A chronological split within 171 days leaves a test period of ~2 months \u2014 too short for a stable estimate and containing no activity-phase diversity.\n\n**Constrains claims.** Combined-instrument results cannot be generalised beyond a single ~6-month solar-maximum window. **The ISRO brief's combined requirement is best served by an ablation on the shared window**, not by a primary benchmark, since only a matched-window comparison attributes a difference to the instrument rather than the epoch."
      },
      {
        "heading": "L-5 \u2014 Spectral features carry little marginal information; hardness ratios carry none for prediction",
        "body": "**Measurement** (55,710 minutes, 40 days): band sums add **\u2264 0.012 AUC** over the total rate (hard 0.8911 vs total 0.8794). Hardness ratios: **\u22120.030** for nowcast, and **0.5075 \u2014 indistinguishable from random \u2014 for 60-min prediction**.\n\n**Constrains features.** Elaborate spectral engineering is **not** justified in advance. Hardness-ratio features for prediction are directly refuted. The 340-channel spectrum must earn inclusion through a measured ablation.\n\n**Constrains claims.** Until the Stage-4 ablation runs, **no claim may be made that spectral resolution improves flare detection or prediction**. The univariate evidence points the other way.\n\n*Scope of this limitation:* it establishes that simple band-sums and ratios add little. It does not prove the full 340-dim spectrum is uninformative under a multivariate model \u2014 which is exactly why the ablation exists."
      },
      {
        "heading": "L-6 \u2014 Class imbalance and its interaction with event scarcity",
        "body": "**Measurement.** Base rates: M/X nowcast **2.37 %**, M/X \u226460 min **4.58 %**, \u2265C nowcast 12.25 %, \u2265C \u226460 min 29.93 %.\n\n**Constrains loss functions.** At 2.4 % positives, unweighted accuracy is meaningless and plain cross-entropy under-weights positives. Class weighting or focal loss is appropriate \u2014 **but** with 581 events, aggressive re-weighting increases variance on the rare class. **Recommendation: mild class weighting, and select on a threshold-free ranking metric (AUC/AP) rather than on accuracy.**\n\n**Constrains evaluation.** Report the **precision\u2013recall frontier**, not a single operating point. v1's most important operational lesson was that a fixed false-alarm budget can bury a working detector; the frontier makes the trade-off visible instead of pre-judged."
      },
      {
        "heading": "L-7 \u2014 Missingness must be masked, never imputed",
        "body": "**Measurement.** T1 `counts_total` NaN on **4.20 %** of minutes; `rate_total` non-finite on **7.6 %**; every NaN flagged (`q_no_data`, `q_partial`). T3's 75 % NaN is **structural** (long form), not missing data.\n\n**Constrains model choice.** Architectures requiring dense complete input need an explicit **mask channel**; imputation is prohibited by dataset policy and would fabricate observations. If an architecture cannot accept masks, prefer a different architecture rather than imputing.\n\n**Constrains evaluation.** The dropped-window fraction is an experimental parameter and must be reported \u2014 dropping differs systematically between quiet and active periods, so silently dropping could bias the base rate."
      },
      {
        "heading": "L-8 \u2014 GTI semantics are not fully characterised",
        "body": "**Measurement.** On **70 of 414 days**, GTI excludes seconds that nevertheless carry finite counts \u2014 **266,919 s** total, up to 43,199 s in one day, concentrated (top-10 days = 80.2 %). Mechanism unknown (A-14 / F-2).\n\n**Constrains features.** `live_time_s` and `gti_fraction` are usable as observability covariates but **their exact semantics are open**. Do not build a physical rate normalisation that depends on GTI meaning more than \"seconds the archive marks good.\"\n\n**Constrains claims.** Exposure-normalised quantities carry an unquantified systematic on those 70 days. Any result sensitive to exposure normalisation should be checked with and without them."
      },
      {
        "heading": "L-9 \u2014 Labels are exogenous and instrument-mismatched",
        "body": "**Measurement.** All labels come from the GOES catalogue, defined on **GOES 1\u20138 \u00c5 flux**. SoLEXS has no cross-calibration to that scale. v1 measured C-class detection at 0.244 recall versus 0.909 for M \u2014 i.e. many catalogued C flares are below SoLEXS's effective sensitivity.\n\n**Constrains claims.** A \"missed\" flare may be a **genuinely unobservable** one, not a model failure. Recall against the GOES catalogue conflates model performance with instrument sensitivity. **Report per-class recall with event counts always attached**, and treat \u2265C results as sensitivity-limited.\n\n**Constrains loss functions.** Training on \u2265C labels injects label noise concentrated in the positive class \u2014 the most damaging kind. **Recommendation: train the primary benchmark on M/X labels** (0.954 vs 0.800 univariate AUC quantifies the difference) and use \u2265C as a robustness check."
      },
      {
        "heading": "L-10 \u2014 Single detector, single solar-cycle phase",
        "body": "**Measurement.** SoLEXS science is **SDD2 only** (SDD1 F-12 inactive across 426 GTI files). The span sits near solar maximum; M/X events per quarter range from **3** (2025Q2) to **181** (2024Q2).\n\n**Constrains validation.** No detector-redundancy check is possible for SoLEXS. The 60\u00d7 quarterly variation means a naive chronological split can produce train/test periods with wildly different base rates \u2014 the split must be chosen with event counts in view (hence the recommended 2024Q1\u20132025Q4 / 2026Q1\u2013Q2 split, ~423 / ~155 events).\n\n**Constrains claims.** **No cross-cycle generalisation claim is supportable.** Results apply to a solar-maximum regime observed by one detector.\n\n---"
      },
      {
        "heading": "Summary \u2014 what can and cannot be claimed",
        "body": "**Supportable with this dataset**\n- M/X flare **nowcast/detection** performance, event-level, with CIs \u2014 the signal is strong and verified (AUC 0.954 from one raw column).\n- **Improvement over persistence** for short-horizon prediction, *if* demonstrated against the mandatory baselines.\n- **Ablation results**: does spectral resolution help? does HEL1OS help? \u2014 as measured deltas with CIs.\n- **Operational characterisation**: recall, false-alarm attribution, latency, the full frontier.\n\n**Not supportable**\n- Calibrated flux or severity regression (**no RMF**).\n- Raw forecasting AUC presented as precursor skill (**horizon-flat**).\n- Standalone X-class conclusions (**47 events**).\n- Cross-cycle or multi-detector generalisation (**one phase, one detector**).\n- Any claim in keV or physical spectral units (**ordinal channels only**).\n\n**The through-line:** this dataset's strength is *detection*, and its strength there is genuine and large. Its forecasting signal is real but is an activity-state effect, and its severity axis is physically unavailable. A programme that leads with detection, controls forecasting against persistence, and treats spectra and HEL1OS as measured ablations will produce defensible results. One that leads with forecasting AUC or severity regression will produce claims this dataset cannot support."
      }
    ],
    "source": "artifacts/v2/ml/DATASET_LIMITATIONS_FOR_ML.md"
  }
}