ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA
Processing flow
Pipeline
From raw archive products to validated scientific evidence. Each stage below produced something that is published on this site and can be checked against it.
What is this?
The processing flow from raw ISSDC archive products to the validated evidence published on this site.
Why should I trust it?
Each stage below produced an artifact that is committed to the repository and rendered elsewhere on this site, so a stage's claim can be checked against its output.
Where can I verify it?
- The same flow, drawnSix architectural views
- What came out of itThe dataset card
- Whether it re-runsDeterminism and integrity
- The stage reportsDigest-addressed JSON
Stages
01
ISSDC archive
Raw Level-1 FITS products for SoLEXS and HEL1OS, as published.
→ Compressed archive files
02
Extraction
Archives unpacked and inventoried. Inactive detectors, malformed GTI, and operating-system sidecar files are identified here rather than downstream.
→ Verified file set
03
Parsing
FITS products read under a frozen contract. Twenty fail-loud rules terminate the parse rather than emit a value the specification cannot vouch for.
→ Typed records
04
Version resolution
Where the archive carries several versions of the same observation minute, a deterministic rule set selects one and records why.
→ One row per (minute, detector)
05
Canonical build
Records written to seven partitioned tables with per-row provenance.
→ 7 tables
06
Validation
Scientific checks against the assembled dataset. Six falsifications of the written specification were found and adjudicated here.
→ Adjudication record
07
Freeze
The dataset is hashed and closed. Every figure on this site descends from that hash.
→ Immutable dataset
What the archive actually contained
Defects found in the source data, recorded rather than quietly worked around.
1. Summary
| Anomaly class | Rule | Count | Products affected | Disposition |
|---|---|---|---|---|
GTI interval STOP ≤ START |
F-19 | 12 | SoLEXS daily archives | Archive defect (CONTRADICTION-005 Defect C) — excluded, logged |
Duplicate HK mjd timestamp |
F-16 | 2 | HEL1OS orbits | Archive defect (CONTRADICTION-006 Defect B) — excluded, logged |
| Inactive detector (empty GTI) | F-12 | 426 | SoLEXS SDD1 GTI files | Legal state, not an error — SDD1 is GTI-only, SDD2 carries science |
| HK time inversions (jitter) | — | see A-12 | HEL1OS orbits | Characterised in MILESTONE_VIII_VALIDATION_REPORT.md (A-12) |
| Two PHA channel families | F-07→resolved | 0 (resolved) | HEL1OS CZT/CdTe | Contract corrected (CONTRADICTION-005 Defect A, r5) — 0 residual failures |
AppleDouble ._* sidecars |
F-18 | (excluded) | filesystem artifacts | exFAT/macOS artifacts, never data — excluded by allowlist |
Total products excluded from the canonical dataset: 14 (12 SoLEXS + 2 HEL1OS), of 436 + 391 = 827 available. Coverage: SoLEXS 424/436, HEL1OS 389/391.
2. F-19 — SoLEXS GTI STOP ≤ START (12 archives)
A GTI interval whose stop time is at or before its start time. STOP ≤ START is not a convention question; it is an internal inconsistency in the archived GTI table. F-19 terminates on parse.
Affected (all SoLEXS daily archives):
AL1_SLX_L1_20240422_v1.0, 20240501, 20250729, 20251205, 20260406, 20260407, 20260408, 20260409, 20260411, 20260413, 20260414, 20260415.
Disposition: archive defect. Excluded from the canonical dataset, individually logged. No amendment (F-19 behaves as designed). Note the temporal clustering (8 of 12 in April 2026) — recorded as an observation, not interpreted.
3. F-16 — Duplicate HK mjd (2 orbits)
Two housekeeping rows sharing an identical mjd timestamp. §2.8 (r4) requires HK mjd to be unique; a duplicate terminates via F-16.
Affected (HEL1OS orbits):
HLS_20260201_120005_43198sec_lev1_V111— 2 duplicated values (of 63,298 rows)HLS_20260202_000005_43183sec_lev1_V111— 1 duplicated value
Disposition: archive defect (CONTRADICTION-006 Defect B, owner ruled not approved for amendment — F-16 continues to terminate). Excluded, logged. 389 of 391 orbits have fully unique HK mjd.
4. F-12 — Inactive detectors (426 GTI files)
A GTI table with zero rows (NAXIS2 = 0), which the contract treats as a legal inactive-detector state, not an error (the single deliberately non-terminating rule). OBSERVED: SoLEXS SDD1 supplies GTI only (no .lc/.pi), and its GTI is empty across the archive; SDD2 carries all science.
Count: 426 F-12 detections among 850 parsed SoLEXS GTI files (SDD1 across the built days). Disposition: expected structural property of the SoLEXS archive; not a defect.
5. Two PHA channel families (resolved, 0 residual)
OBSERVED during the archive-wide build: HEL1OS has two detector families with different PHA channel counts — CZT = 341, CdTe = 511. The r0 contract asserted a single DETCHANS = 341, which blocked all 391 orbits at F-07.
Disposition: contract corrected (CONTRADICTION-005 Defect A → spec r5: per-family allowlist; F-11 restated over three incommensurable channel spaces — SoLEXS PI 340, HEL1OS CZT PHA 341, HEL1OS CdTe PHA 511). After the fix: 0 F-07 DETCHANS failures archive-wide. Not an archive defect — an archive property the contract now encodes.
6. AppleDouble sidecars (._*)
macOS writes ._<name> metadata sidecars on the exFAT volume. These are filesystem artifacts, never instrument data. F-18 (allowlist + ._* guard) excludes them. Disposition: not data; excluded structurally. (Note: the withdrawn "10 F-01 unreadable days" figure from CONTRADICTION-005 originated in an ad-hoc scan that globbed these sidecars; the authoritative build has no F-01 skips.)
artifacts/v2/phase05/ARCHIVE_QUALITY_REPORT.md
Canonical dataset profile
1. Row counts
| Table | Files | Rows |
|---|---|---|
T1 solexs_lc_1min |
424 | 610,560 |
T2 solexs_spec_1min |
424 | 610,560 |
T3 hel1os_lc_1min |
373 | 1,027,773 |
T4 hel1os_hk_1min |
389 | 277,054 |
T5 hel1os_spec_1min |
373 | 1,026,816 |
T6 gti_intervals |
1 | 2,130 |
T7 provenance_manifest |
1 | 5,199 |
2. Date coverage
| Table | First | Last |
|---|---|---|
| T1/T2 (SoLEXS) | 2024-02-01 00:00:00+00:00 | 2026-06-15 23:59:00+00:00 |
| T3 (HEL1OS LC) | 2025-12-07 00:00:00+00:00 | 2026-06-17 23:59:00+00:00 |
| T4 (HEL1OS HK) | 2025-12-07 00:00:00+00:00 | 2026-06-17 23:59:00+00:00 |
3. Missing-value statistics
- T1
counts_total: 25,623 NaN of 610,560 rows (4.20%).q_no_dataminutes: 25,623;q_partial: 2,063. - T2:
q_no_dataminutes: 25,623 of 610,560. Channel arrays preserve NaN per the r2/r5 rules; never imputed. - T3 band-rate cells: 15,418,723 NaN of 20,555,460 (75.01%) across 20 band columns.
- No imputation, filling, or interpolation exists anywhere in the pipeline (verified by test suite).
4. GTI coverage (T6)
- Intervals: 2,130 ({'solexs': 2130})
- Total good exposure: 33829521 s (391.5 days)
- Interval duration: min 2.0 s | median 4363.0 s | mean 15882.4 s | max 59916.0 s
- SoLEXS detector activity: SDD2 active 425 days; SDD1 inactive (F-12) on 426 of 850 parsed GTI files.
- T1 live time total: 33829521 s. GTI-excluded seconds carrying finite counts are counted per day and [A-14] remain unexplained — Milestone VIII.
5. Version-resolution statistics
- Orbit files examined: 391
- Owned (minute, detector) pairs: 1,065,572
- Conflicting pairs: 48,604 in 256 distinct conflicts
- Resolved by R1 (higher version): 47,328
- Resolved by R2 (longer duration): 1,276
- R3 (processing date) invoked: 0 — HEL1OS primaries carry no
DATEheader - F-14 terminations: 0
- Unique ownership: holds — every owned pair has exactly one provenance owner (asserted at map construction and by F-15 output guards).
6. Provenance completeness (T7)
- Provenance rows: 5,199; duplicate (file, product, detector) rows: 0
- Output rows missing provenance: {'T1': 0, 'T3': 0, 'T4': 0}
- Orphan rows (src_file absent from T7): {'T1': 0, 'T3': 0, 'T4': 0}
artifacts/v2/phase05/CANONICAL_DATASET_PROFILE.md
Table schemas
T1 — solexs_lc_1min (610,560 rows · 424 files)
One row per minute of a SoLEXS observing day. Source: .lc (light curve) + .gti, SDD2.
| Column | Type | Units | Source / derivation | Allowed values | Missing semantics |
|---|---|---|---|---|---|
timestamp |
datetime64[ns,UTC] | — | minute index of the observation day | 1,440/day, monotonic, unique | never missing |
counts_total |
float64 | counts | Σ of .lc COUNTS over finite seconds in the minute |
≥ 0 | NaN ⟺ no finite second (q_no_data) |
live_time_s |
float64 | s | GTI-covered seconds in the minute (inclusive endpoints, STOP−START+1) |
0 … 60 | never missing; 0 = fully excluded |
rate_total |
float64 | counts/s | counts_total / live_time_s |
≥ 0 | NaN where live_time_s == 0 — not imputed |
gti_fraction |
float64 | — | live_time_s / 60 |
0.0 … 1.0 | never missing |
n_seconds_present |
int16 | count | number of finite .lc seconds in the minute |
0 … 60 | never missing |
q_no_data |
bool | — | n_seconds_present == 0 |
True/False | flag |
q_partial |
bool | — | 0 < gti_fraction < 1 |
True/False | flag |
detector |
category | — | SoLEXS detector | SDD2 only |
never missing |
src_file |
object | — | source .lc path |
— | never missing |
src_sha256 |
object | — | SHA-256 of the source archive (Phase 0.5.1 manifest) | 64 hex | never missing |
archive_version |
object | — | ISSDC archive variant | v1.0, v1.1 |
never missing |
counts_total(light curve) is band-limited and is not Σ(T2counts) — seeSCIENTIFIC_FINDINGS.mdF-1. The two are carried independently; neither is derived from the other.
T2 — solexs_spec_1min (610,560 rows · 424 files)
One row per minute; the full SoLEXS spectrum. Source: .pi (OGIP Type II PHA) + .gti, SDD2.
| Column | Type | Units | Source / derivation | Allowed values | Missing semantics |
|---|---|---|---|---|---|
timestamp |
datetime64[ns,UTC] | — | minute index | 1,440/day | never missing |
counts |
object → array[340] float64 | counts | per-channel Σ over finite seconds in the minute | ≥ 0 per channel | NaN per channel where no finite second |
live_time_s |
float64 | s | GTI-covered seconds (inclusive) | 0 … 60 | never missing |
gti_fraction |
float64 | — | live_time_s / 60 |
0.0 … 1.0 | never missing |
n_seconds_present |
int16 | count | finite spectra contributing | 0 … 60 | never missing |
q_no_data |
bool | — | no finite spectrum in the minute | True/False | flag |
q_partial |
bool | — | partial GTI coverage | True/False | flag |
detector |
category | — | SoLEXS detector | SDD2 |
never missing |
chantype |
category | — | channel type from CHANTYPE |
PI |
never missing |
src_file, src_sha256, archive_version |
object | — | provenance | — | never missing |
Channel axis: counts[i] is PI channel i, i ∈ [0, 339] — an ordinal index. No channel_energy_keV column exists, by design: the archive contains no response matrix, so no channel→energy mapping is possible. Do not assume linearity in energy.
T3 — hel1os_lc_1min (1,027,773 rows · 373 files)
LONG form (spec r6): one row per (minute, detector), keyed uniquely by that pair. Source: HEL1OS lightcurve_*.fits, ownership from the Version Resolution Engine.
| Column | Type | Units | Source / derivation | Allowed values | Missing semantics |
|---|---|---|---|---|---|
timestamp |
datetime64[ns,UTC] | — | minute index | — | never missing |
detector |
object | — | the detector this row describes | CZT1,CZT2,CDTE1,CDTE2 |
never missing |
{det}_{lo}_{hi}_rate |
float64 | counts/s | mean of CTR over the minute's samples |
≥ 0 | NaN when {det} ≠ this row's detector (structural) |
{det}_{lo}_{hi}_stat_err |
float64 | counts/s | mean of STAT_ERR |
≥ 0 | as above |
{det}_{lo}_{hi}_n_samples |
float64 | count | samples contributing to the minute | ≥ 1 | as above |
orbit_id |
object | — | owning orbit (version-resolved) | HLS_…_V### |
never missing |
orbit_version |
object | — | orbit product version | V111,V211,V112,V311 |
never missing |
src_file, src_sha256, archive_version |
object | — | provenance | — | never missing |
Band columns present (edges parsed from EXTNAME, allowlist-validated):
- CZT1/CZT2:
20_40,40_60,60_80,80_150,18_160(keV) - CDTE1/CDTE2:
5_20,20_30,30_40,40_60,1p8_90(keV;1p8= 1.8)
Structural NaN: because the table is long-form, each row populates only its own detector's band columns; the other three detectors' columns are NaN by construction, not by missing data. Filter by
detectorbefore using band columns. A wide view may be derived, but the canonical form is long (spec r6) because ownership is per (minute, detector).
T4 — hel1os_hk_1min (277,054 rows · 389 files)
One row per minute of HEL1OS housekeeping. Source: aux/hk.fits.
| Column | Type | Units | Derivation | Notes |
|---|---|---|---|---|
timestamp |
datetime64[ns,UTC] | — | minute index | never missing |
czt1temp, czt2temp, cdte1temp, cdte2temp |
float64 | °C | minute mean | detector temperatures |
czthvmon, cdtehvmon |
float64 | V | minute mean | high-voltage monitors |
czt1ctr, czt2ctr, cdte1ctr, cdte2ctr |
float64 | counts/s | minute mean | per-detector rate monitors |
czt1enth |
float64 | keV | minute mean | energy threshold |
cdte1enerthr, cdte2enerthr |
float64 | (undeclared) | minute mean | energy thresholds; unit not declared in the archive |
cdte1pilectr, cdte2pilectr |
float64 | count | minute max | pile-up counters |
czt1satctr1, czt2satctr1 |
float64 | count | minute max | saturation counters |
czt1hotpixcnt, czt2hotpixcnt |
float64 | count | minute max | hot-pixel counters |
sunradeg, sundecdeg |
float64 | degrees | minute mean | Sun pointing |
fehkstat |
float64 | — | minute mean | front-end housekeeping status word |
suninfov |
bool | — | minute min (conservative: True only if every sample is True) | Sun-in-field-of-view |
n_samples |
int64 | count | HK records in the minute | ≥ 1 |
orbit_id, src_file, src_sha256, archive_version |
object | — | provenance | never missing |
Ordering: the parser preserves archive (telemetry-arrival) order and never sorts. Source records contain sub-second out-of-order steps (
SCIENTIFIC_FINDINGS.mdF-3). All aggregations used here are order-independent. A consumer needing chronological order must callapp.v2.utils.timeseries.chronological_sort()explicitly.
T5 — hel1os_spec_1min (1,026,816 rows · 373 files)
One row per (minute, detector); HEL1OS spectra. Source: hel1os_*_spectra_*.fits.
| Column | Type | Units | Derivation | Allowed values | Missing semantics |
|---|---|---|---|---|---|
timestamp |
datetime64[ns,UTC] | — | minute index | — | never missing |
detector |
object | — | spectra detector | CZT1,CZT2,CDTE1,CDTE2 |
never missing |
counts |
object → array[detchans] float64 |
counts | per-channel Σ over finite spectra in the minute | ≥ 0 | NaN per channel where absent |
stat_err |
object → array[detchans] float64 |
counts | per-channel Σ of STAT_ERR |
≥ 0 | as above |
live_time_s |
float64 | s | Σ EXPOSURE of contributing spectra |
≥ 0 | never missing |
n_spectra |
int64 | count | spectra contributing to the minute | ≥ 1 | never missing |
chantype |
category | — | from CHANTYPE |
PHA |
never missing |
detchans |
int16 | count | channel-space size — carried explicitly (r5) | 341 (CZT) or 511 (CdTe) | never missing |
orbit_id, src_file, src_sha256, archive_version |
object | — | provenance | — | never missing |
Three incommensurable channel spaces (F-4): SoLEXS PI 340, HEL1OS CZT PHA 341, HEL1OS CdTe PHA 511. Never concatenate
countsarrays across families or instruments — always filter bydetchans/detectorfirst. Channel indices are ordinal; no response matrix exists.
T6 — gti_intervals (2,130 rows · 1 file)
Long-form good-time intervals, one row per interval.
| Column | Type | Units | Derivation | Notes |
|---|---|---|---|---|
instrument |
object | — | source instrument | solexs |
detector |
object | — | detector | SDD2 (SDD1 contributes no intervals — F-12 inactive) |
start_utc |
datetime64[ns,UTC] | — | interval start | inclusive |
stop_utc |
datetime64[ns,UTC] | — | interval stop | inclusive |
duration_s |
float64 | s | stop − start + 1 (inclusive convention) |
> 0 always |
src_file, src_sha256 |
object | — | provenance | never missing |
A GTI-excluded second may carry a finite count (spec r5). The invariant is one-directional:
NaN(counts) ⇒ GTI-excluded. The excess is unexplained (A-14 / F-2).
T7 — provenance_manifest (5,199 rows · 1 file)
One row per parsed source file. Every canonical row traces here via src_file.
| Column | Type | Derivation | Notes |
|---|---|---|---|
src_file |
object | source path | join key; unique with (product,detector) |
src_sha256 |
object | archive SHA-256 (Phase 0.5.1 manifest) | 64 hex |
instrument |
object | solexs / hel1os |
— |
detector |
object | e.g. SDD1,SDD2,CZT1,CDTE2 |
None for whole-orbit products |
product |
object | lc,pi,gti,hk,spectra,lightcurve,events |
— |
archive_version |
object | v1.0,v1.1,V111,… |
— |
obs_date |
object | YYYYMMDD |
— |
orbit_id |
object | HEL1OS orbit, else None |
— |
creator |
object | producing pipeline, from the FITS header | e.g. solexs_pipeline-1.4 |
processing_date |
object | header DATE |
None for HEL1OS (no DATE header) |
rows_in / rows_out |
int64 | source rows / emitted rows | 0/0 legal for F-12 inactive detectors |
parser_version |
object | parser identity | v2-0.5.2 |
parsed_at_utc |
object | build wall-clock time | the one non-reproducible field — excluded from content hashes |
time_epoch_resolution |
object | resolved time convention | unix_seconds (SoLEXS) / relative_seconds (HEL1OS spectra) / mjd_days |
assumptions_applied |
object → array[str] | assumptions recorded at parse | audit trail |
Provenance behaviour (all tables)
- Every canonical row carries
src_file+src_sha256; no row lacks provenance (verified: 0 orphans, 0 duplicates, 0 missing). src_sha256is the archive SHA-256 from the Phase 0.5.1 manifest — it pins the input, not the output.- HEL1OS tables additionally carry
orbit_id, resolved by the Version Resolution Engine per (minute, detector). Two detectors in the same minute may legitimately have differentorbit_ids at orbit boundaries. - Provenance columns are not measurements; they must not be used as features.
artifacts/v2/phase05/DATA_DICTIONARY.md