Skip to content

ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA

Processing flow

Pipeline

From raw archive products to validated scientific evidence. Each stage below produced something that is published on this site and can be checked against it.

What is this?

The processing flow from raw ISSDC archive products to the validated evidence published on this site.

Why should I trust it?

Each stage below produced an artifact that is committed to the repository and rendered elsewhere on this site, so a stage's claim can be checked against its output.

Where can I verify it?

Stages

  1. 01

    ISSDC archive

    Raw Level-1 FITS products for SoLEXS and HEL1OS, as published.

    → Compressed archive files

  2. 02

    Extraction

    Archives unpacked and inventoried. Inactive detectors, malformed GTI, and operating-system sidecar files are identified here rather than downstream.

    → Verified file set

  3. 03

    Parsing

    FITS products read under a frozen contract. Twenty fail-loud rules terminate the parse rather than emit a value the specification cannot vouch for.

    → Typed records

  4. 04

    Version resolution

    Where the archive carries several versions of the same observation minute, a deterministic rule set selects one and records why.

    → One row per (minute, detector)

  5. 05

    Canonical build

    Records written to seven partitioned tables with per-row provenance.

    → 7 tables

  6. 06

    Validation

    Scientific checks against the assembled dataset. Six falsifications of the written specification were found and adjudicated here.

    → Adjudication record

  7. 07

    Freeze

    The dataset is hashed and closed. Every figure on this site descends from that hash.

    → Immutable dataset

What the archive actually contained

Defects found in the source data, recorded rather than quietly worked around.

1. Summary

Anomaly class Rule Count Products affected Disposition
GTI interval STOP ≤ START F-19 12 SoLEXS daily archives Archive defect (CONTRADICTION-005 Defect C) — excluded, logged
Duplicate HK mjd timestamp F-16 2 HEL1OS orbits Archive defect (CONTRADICTION-006 Defect B) — excluded, logged
Inactive detector (empty GTI) F-12 426 SoLEXS SDD1 GTI files Legal state, not an error — SDD1 is GTI-only, SDD2 carries science
HK time inversions (jitter) see A-12 HEL1OS orbits Characterised in MILESTONE_VIII_VALIDATION_REPORT.md (A-12)
Two PHA channel families F-07→resolved 0 (resolved) HEL1OS CZT/CdTe Contract corrected (CONTRADICTION-005 Defect A, r5) — 0 residual failures
AppleDouble ._* sidecars F-18 (excluded) filesystem artifacts exFAT/macOS artifacts, never data — excluded by allowlist

Total products excluded from the canonical dataset: 14 (12 SoLEXS + 2 HEL1OS), of 436 + 391 = 827 available. Coverage: SoLEXS 424/436, HEL1OS 389/391.

2. F-19 — SoLEXS GTI STOP ≤ START (12 archives)

A GTI interval whose stop time is at or before its start time. STOP ≤ START is not a convention question; it is an internal inconsistency in the archived GTI table. F-19 terminates on parse.

Affected (all SoLEXS daily archives): AL1_SLX_L1_20240422_v1.0, 20240501, 20250729, 20251205, 20260406, 20260407, 20260408, 20260409, 20260411, 20260413, 20260414, 20260415.

Disposition: archive defect. Excluded from the canonical dataset, individually logged. No amendment (F-19 behaves as designed). Note the temporal clustering (8 of 12 in April 2026) — recorded as an observation, not interpreted.

3. F-16 — Duplicate HK mjd (2 orbits)

Two housekeeping rows sharing an identical mjd timestamp. §2.8 (r4) requires HK mjd to be unique; a duplicate terminates via F-16.

Affected (HEL1OS orbits):

  • HLS_20260201_120005_43198sec_lev1_V111 — 2 duplicated values (of 63,298 rows)
  • HLS_20260202_000005_43183sec_lev1_V111 — 1 duplicated value

Disposition: archive defect (CONTRADICTION-006 Defect B, owner ruled not approved for amendment — F-16 continues to terminate). Excluded, logged. 389 of 391 orbits have fully unique HK mjd.

4. F-12 — Inactive detectors (426 GTI files)

A GTI table with zero rows (NAXIS2 = 0), which the contract treats as a legal inactive-detector state, not an error (the single deliberately non-terminating rule). OBSERVED: SoLEXS SDD1 supplies GTI only (no .lc/.pi), and its GTI is empty across the archive; SDD2 carries all science.

Count: 426 F-12 detections among 850 parsed SoLEXS GTI files (SDD1 across the built days). Disposition: expected structural property of the SoLEXS archive; not a defect.

5. Two PHA channel families (resolved, 0 residual)

OBSERVED during the archive-wide build: HEL1OS has two detector families with different PHA channel counts — CZT = 341, CdTe = 511. The r0 contract asserted a single DETCHANS = 341, which blocked all 391 orbits at F-07.

Disposition: contract corrected (CONTRADICTION-005 Defect A → spec r5: per-family allowlist; F-11 restated over three incommensurable channel spaces — SoLEXS PI 340, HEL1OS CZT PHA 341, HEL1OS CdTe PHA 511). After the fix: 0 F-07 DETCHANS failures archive-wide. Not an archive defect — an archive property the contract now encodes.

6. AppleDouble sidecars (._*)

macOS writes ._<name> metadata sidecars on the exFAT volume. These are filesystem artifacts, never instrument data. F-18 (allowlist + ._* guard) excludes them. Disposition: not data; excluded structurally. (Note: the withdrawn "10 F-01 unreadable days" figure from CONTRADICTION-005 originated in an ad-hoc scan that globbed these sidecars; the authoritative build has no F-01 skips.)

artifacts/v2/phase05/ARCHIVE_QUALITY_REPORT.md

Canonical dataset profile

1. Row counts

Table Files Rows
T1 solexs_lc_1min 424 610,560
T2 solexs_spec_1min 424 610,560
T3 hel1os_lc_1min 373 1,027,773
T4 hel1os_hk_1min 389 277,054
T5 hel1os_spec_1min 373 1,026,816
T6 gti_intervals 1 2,130
T7 provenance_manifest 1 5,199

2. Date coverage

Table First Last
T1/T2 (SoLEXS) 2024-02-01 00:00:00+00:00 2026-06-15 23:59:00+00:00
T3 (HEL1OS LC) 2025-12-07 00:00:00+00:00 2026-06-17 23:59:00+00:00
T4 (HEL1OS HK) 2025-12-07 00:00:00+00:00 2026-06-17 23:59:00+00:00

3. Missing-value statistics

  • T1 counts_total: 25,623 NaN of 610,560 rows (4.20%). q_no_data minutes: 25,623; q_partial: 2,063.
  • T2: q_no_data minutes: 25,623 of 610,560. Channel arrays preserve NaN per the r2/r5 rules; never imputed.
  • T3 band-rate cells: 15,418,723 NaN of 20,555,460 (75.01%) across 20 band columns.
  • No imputation, filling, or interpolation exists anywhere in the pipeline (verified by test suite).

4. GTI coverage (T6)

  • Intervals: 2,130 ({'solexs': 2130})
  • Total good exposure: 33829521 s (391.5 days)
  • Interval duration: min 2.0 s | median 4363.0 s | mean 15882.4 s | max 59916.0 s
  • SoLEXS detector activity: SDD2 active 425 days; SDD1 inactive (F-12) on 426 of 850 parsed GTI files.
  • T1 live time total: 33829521 s. GTI-excluded seconds carrying finite counts are counted per day and [A-14] remain unexplained — Milestone VIII.

5. Version-resolution statistics

  • Orbit files examined: 391
  • Owned (minute, detector) pairs: 1,065,572
  • Conflicting pairs: 48,604 in 256 distinct conflicts
  • Resolved by R1 (higher version): 47,328
  • Resolved by R2 (longer duration): 1,276
  • R3 (processing date) invoked: 0 — HEL1OS primaries carry no DATE header
  • F-14 terminations: 0
  • Unique ownership: holds — every owned pair has exactly one provenance owner (asserted at map construction and by F-15 output guards).

6. Provenance completeness (T7)

  • Provenance rows: 5,199; duplicate (file, product, detector) rows: 0
  • Output rows missing provenance: {'T1': 0, 'T3': 0, 'T4': 0}
  • Orphan rows (src_file absent from T7): {'T1': 0, 'T3': 0, 'T4': 0}

artifacts/v2/phase05/CANONICAL_DATASET_PROFILE.md

Table schemas

T1 — solexs_lc_1min (610,560 rows · 424 files)

One row per minute of a SoLEXS observing day. Source: .lc (light curve) + .gti, SDD2.

Column Type Units Source / derivation Allowed values Missing semantics
timestamp datetime64[ns,UTC] minute index of the observation day 1,440/day, monotonic, unique never missing
counts_total float64 counts Σ of .lc COUNTS over finite seconds in the minute ≥ 0 NaN ⟺ no finite second (q_no_data)
live_time_s float64 s GTI-covered seconds in the minute (inclusive endpoints, STOP−START+1) 0 … 60 never missing; 0 = fully excluded
rate_total float64 counts/s counts_total / live_time_s ≥ 0 NaN where live_time_s == 0not imputed
gti_fraction float64 live_time_s / 60 0.0 … 1.0 never missing
n_seconds_present int16 count number of finite .lc seconds in the minute 0 … 60 never missing
q_no_data bool n_seconds_present == 0 True/False flag
q_partial bool 0 < gti_fraction < 1 True/False flag
detector category SoLEXS detector SDD2 only never missing
src_file object source .lc path never missing
src_sha256 object SHA-256 of the source archive (Phase 0.5.1 manifest) 64 hex never missing
archive_version object ISSDC archive variant v1.0, v1.1 never missing

counts_total (light curve) is band-limited and is not Σ(T2 counts) — see SCIENTIFIC_FINDINGS.md F-1. The two are carried independently; neither is derived from the other.

T2 — solexs_spec_1min (610,560 rows · 424 files)

One row per minute; the full SoLEXS spectrum. Source: .pi (OGIP Type II PHA) + .gti, SDD2.

Column Type Units Source / derivation Allowed values Missing semantics
timestamp datetime64[ns,UTC] minute index 1,440/day never missing
counts object → array[340] float64 counts per-channel Σ over finite seconds in the minute ≥ 0 per channel NaN per channel where no finite second
live_time_s float64 s GTI-covered seconds (inclusive) 0 … 60 never missing
gti_fraction float64 live_time_s / 60 0.0 … 1.0 never missing
n_seconds_present int16 count finite spectra contributing 0 … 60 never missing
q_no_data bool no finite spectrum in the minute True/False flag
q_partial bool partial GTI coverage True/False flag
detector category SoLEXS detector SDD2 never missing
chantype category channel type from CHANTYPE PI never missing
src_file, src_sha256, archive_version object provenance never missing

Channel axis: counts[i] is PI channel i, i ∈ [0, 339] — an ordinal index. No channel_energy_keV column exists, by design: the archive contains no response matrix, so no channel→energy mapping is possible. Do not assume linearity in energy.

T3 — hel1os_lc_1min (1,027,773 rows · 373 files)

LONG form (spec r6): one row per (minute, detector), keyed uniquely by that pair. Source: HEL1OS lightcurve_*.fits, ownership from the Version Resolution Engine.

Column Type Units Source / derivation Allowed values Missing semantics
timestamp datetime64[ns,UTC] minute index never missing
detector object the detector this row describes CZT1,CZT2,CDTE1,CDTE2 never missing
{det}_{lo}_{hi}_rate float64 counts/s mean of CTR over the minute's samples ≥ 0 NaN when {det} ≠ this row's detector (structural)
{det}_{lo}_{hi}_stat_err float64 counts/s mean of STAT_ERR ≥ 0 as above
{det}_{lo}_{hi}_n_samples float64 count samples contributing to the minute ≥ 1 as above
orbit_id object owning orbit (version-resolved) HLS_…_V### never missing
orbit_version object orbit product version V111,V211,V112,V311 never missing
src_file, src_sha256, archive_version object provenance never missing

Band columns present (edges parsed from EXTNAME, allowlist-validated):

  • CZT1/CZT2: 20_40, 40_60, 60_80, 80_150, 18_160 (keV)
  • CDTE1/CDTE2: 5_20, 20_30, 30_40, 40_60, 1p8_90 (keV; 1p8 = 1.8)

Structural NaN: because the table is long-form, each row populates only its own detector's band columns; the other three detectors' columns are NaN by construction, not by missing data. Filter by detector before using band columns. A wide view may be derived, but the canonical form is long (spec r6) because ownership is per (minute, detector).

T4 — hel1os_hk_1min (277,054 rows · 389 files)

One row per minute of HEL1OS housekeeping. Source: aux/hk.fits.

Column Type Units Derivation Notes
timestamp datetime64[ns,UTC] minute index never missing
czt1temp, czt2temp, cdte1temp, cdte2temp float64 °C minute mean detector temperatures
czthvmon, cdtehvmon float64 V minute mean high-voltage monitors
czt1ctr, czt2ctr, cdte1ctr, cdte2ctr float64 counts/s minute mean per-detector rate monitors
czt1enth float64 keV minute mean energy threshold
cdte1enerthr, cdte2enerthr float64 (undeclared) minute mean energy thresholds; unit not declared in the archive
cdte1pilectr, cdte2pilectr float64 count minute max pile-up counters
czt1satctr1, czt2satctr1 float64 count minute max saturation counters
czt1hotpixcnt, czt2hotpixcnt float64 count minute max hot-pixel counters
sunradeg, sundecdeg float64 degrees minute mean Sun pointing
fehkstat float64 minute mean front-end housekeeping status word
suninfov bool minute min (conservative: True only if every sample is True) Sun-in-field-of-view
n_samples int64 count HK records in the minute ≥ 1
orbit_id, src_file, src_sha256, archive_version object provenance never missing

Ordering: the parser preserves archive (telemetry-arrival) order and never sorts. Source records contain sub-second out-of-order steps (SCIENTIFIC_FINDINGS.md F-3). All aggregations used here are order-independent. A consumer needing chronological order must call app.v2.utils.timeseries.chronological_sort() explicitly.

T5 — hel1os_spec_1min (1,026,816 rows · 373 files)

One row per (minute, detector); HEL1OS spectra. Source: hel1os_*_spectra_*.fits.

Column Type Units Derivation Allowed values Missing semantics
timestamp datetime64[ns,UTC] minute index never missing
detector object spectra detector CZT1,CZT2,CDTE1,CDTE2 never missing
counts object → array[detchans] float64 counts per-channel Σ over finite spectra in the minute ≥ 0 NaN per channel where absent
stat_err object → array[detchans] float64 counts per-channel Σ of STAT_ERR ≥ 0 as above
live_time_s float64 s Σ EXPOSURE of contributing spectra ≥ 0 never missing
n_spectra int64 count spectra contributing to the minute ≥ 1 never missing
chantype category from CHANTYPE PHA never missing
detchans int16 count channel-space size — carried explicitly (r5) 341 (CZT) or 511 (CdTe) never missing
orbit_id, src_file, src_sha256, archive_version object provenance never missing

Three incommensurable channel spaces (F-4): SoLEXS PI 340, HEL1OS CZT PHA 341, HEL1OS CdTe PHA 511. Never concatenate counts arrays across families or instruments — always filter by detchans/detector first. Channel indices are ordinal; no response matrix exists.

T6 — gti_intervals (2,130 rows · 1 file)

Long-form good-time intervals, one row per interval.

Column Type Units Derivation Notes
instrument object source instrument solexs
detector object detector SDD2 (SDD1 contributes no intervals — F-12 inactive)
start_utc datetime64[ns,UTC] interval start inclusive
stop_utc datetime64[ns,UTC] interval stop inclusive
duration_s float64 s stop − start + 1 (inclusive convention) > 0 always
src_file, src_sha256 object provenance never missing

A GTI-excluded second may carry a finite count (spec r5). The invariant is one-directional: NaN(counts) ⇒ GTI-excluded. The excess is unexplained (A-14 / F-2).

T7 — provenance_manifest (5,199 rows · 1 file)

One row per parsed source file. Every canonical row traces here via src_file.

Column Type Derivation Notes
src_file object source path join key; unique with (product,detector)
src_sha256 object archive SHA-256 (Phase 0.5.1 manifest) 64 hex
instrument object solexs / hel1os
detector object e.g. SDD1,SDD2,CZT1,CDTE2 None for whole-orbit products
product object lc,pi,gti,hk,spectra,lightcurve,events
archive_version object v1.0,v1.1,V111,…
obs_date object YYYYMMDD
orbit_id object HEL1OS orbit, else None
creator object producing pipeline, from the FITS header e.g. solexs_pipeline-1.4
processing_date object header DATE None for HEL1OS (no DATE header)
rows_in / rows_out int64 source rows / emitted rows 0/0 legal for F-12 inactive detectors
parser_version object parser identity v2-0.5.2
parsed_at_utc object build wall-clock time the one non-reproducible field — excluded from content hashes
time_epoch_resolution object resolved time convention unix_seconds (SoLEXS) / relative_seconds (HEL1OS spectra) / mjd_days
assumptions_applied object → array[str] assumptions recorded at parse audit trail

Provenance behaviour (all tables)

  1. Every canonical row carries src_file + src_sha256; no row lacks provenance (verified: 0 orphans, 0 duplicates, 0 missing).
  2. src_sha256 is the archive SHA-256 from the Phase 0.5.1 manifest — it pins the input, not the output.
  3. HEL1OS tables additionally carry orbit_id, resolved by the Version Resolution Engine per (minute, detector). Two detectors in the same minute may legitimately have different orbit_ids at orbit boundaries.
  4. Provenance columns are not measurements; they must not be used as features.

artifacts/v2/phase05/DATA_DICTIONARY.md