ILLUSTRATIVE · NASA / SVS · NOT ADITYA-L1 DATA
Engineering · reproducibility
What can be re-run, and what cannot
Six properties, each backed by something that was actually executed — and each stating plainly where it stops. Two of the six are not fully verified, and say so.
What is this?
A reproducibility record for the dataset, the environment and the published evidence: what is pinned, what is digested, what was re-run, and what has never been executed.
Why should I trust it?
Every figure here is read from a committed artifact or computed from the repository's own git history at build time. Where something has not been run, the metric reports it as unverified rather than omitting it.
Where can I verify it?
- The rebuild commandsRun it yourself
- The engineering recordAdjudicated contradictions
- Evidence traceabilityEvery claim, to its bytes
- The raw payloadreproducibility.json
Section 1
Six reproducibility metrics
Not throughput, not latency. This site serves a frozen dataset from static files — it has no ingestion loop and no query service to benchmark, so those numbers would have to be invented.
Environment reproducibility
Partial
99 packages pinned
Python 3.12.12 on macOS-26.5.2-arm64-arm-64bit, with every dependency pinned to an exact version in a committed lockfile. The manifest's recorded digest does not match the file today — that discrepancy is explained in full below, and the explanation is computed rather than asserted.
Artifact integrity
Verified
1,985 / 1,985 files digested
Every Parquet file in the freeze carries its own SHA-256 in the manifest, and the 7 tables roll up to a single dataset digest. A changed byte anywhere changes the digest that this site prints in its header.
Dataset provenance
Verified
row-level source attribution
Every row in every table carries the path and SHA-256 of the archive file it was parsed from, joinable to the provenance manifest table. Provenance is a column, not a document — so it cannot be lost, and it cannot disagree with the data.
Build determinism
Partial
24 / 24 comparisons byte-identical
12 days were rebuilt from the source archive and compared against the frozen output across T1 and T2. Every comparison matched both in content and byte-for-byte. This is a sample, not the full 1,985-file archive.
Validation coverage
Partial
5 of 6 contradictions closed
Every recorded disagreement between the written specification and the implementation is adjudicated in public, including the declined amendments. 1 remains open and is published as open.
Evidence traceability
Verified
6 measurements, 2 artifacts
Every measured quantity rendered on this site resolves to a JSON pointer inside a committed artifact, with the digest of the exact bytes read and the commit that produced them. A CI gate re-reads the artifacts and fails the build on any drift.
Section 2
Case study: the lockfile digest that did not match
The environment record and the environment on disk disagreed. That is the worst thing an integrity system can report — so it was traced rather than silenced.
Recorded in manifest
6899e001b1c4d64a5aec01b3ee1d2cfce2a3ff38ab64f827a75ab9d55fd13d3f
The file today
8187301ecffc0bac6ae2ff97b9e0cb6791d20fdaf8afd31614a1d7cb6ea80320
The file at be0b7e5
6899e001b1c4d64a5aec01b3ee1d2cfce2a3ff38ab64f827a75ab9d55fd13d3f
The resolution. The recorded digest is not stale and it is not wrong. It is the digest of the lockfile as it existed at the dataset freeze commit, and re-hashing that blob out of git reproduces it exactly. What changed afterwards was the file, not the record.
1
Dataset freeze
The canonical dataset is built and frozen at commit be0b7e5.
freeze_manifest.json
2
Frozen lockfile
The manifest records the digest of the environment used: 92 pinned packages.
sha256 6899e001b1c4…
3
Benchmark expansion
Months later, the flare benchmark is run. It needs gradient boosting and SHAP.
benchmark_results.json
4
Additional dependencies
7 packages are appended to the same lockfile.
99 pinned packages
5
Final runtime
The lockfile on disk now describes the benchmark environment, not the freeze environment.
sha256 8187301ecffc…
The 7 packages added after the freeze
- cloudpickle==3.1.2
- lightgbm==4.6.0
- llvmlite==0.48.0
- numba==0.66.0
- shap==0.52.0
- slicer==0.0.8
- tqdm==4.69.0
Gradient boosting, SHAP and their transitive dependencies — exactly the tools the benchmark needed and the dataset build did not. The package set grew from 92 to 99.
The lockfile's entire history
9efad0c
Milestone XI: baseline benchmark — ML provides NO operational value beyond a threshold
be0b7e5
Milestone VIII COMPLETE: all assumptions discharged, CONTRADICTION-003 resolved
Two commits, and the second explains the mismatch entirely.
Why this is expected. A lockfile is a living description of a working environment; a manifest digest is a dated claim about one moment. The two diverge the first time a project does something new with the same data — which for a research repository is not an edge case, it is the normal path.
Why provenance mattered. Without the recorded commit, the only available reading of a digest mismatch is that the environment record is untrustworthy, and the honest response would have been to withdraw the reproducibility claim. Because the manifest records be0b7e5, the question was answerable: re-hash the blob at that commit, diff the package sets, read the two commit subjects. The system produced a correct alarm and then produced its own explanation.
What was not done. The manifest was not quietly re-hashed to make the mismatch disappear. It records what the dataset was built with, which is the question it exists to answer; the benchmark environment is a different question, answered here. An integrity system whose alarms are silenced to keep a page green is decorative.
Section 3
Build determinism
12 days rebuilt from the source archive, compared against the frozen output. Every comparison matched byte for byte.
- Days rebuilt
- 12
- Content matches
- 24
- Byte matches
- 24
- Tables covered
- T1, T2
The limit of this claim. This is a 12-day sample across 2 of 7 tables, not the full 1,985-file archive. Byte-identical Parquet output is a strong signal — it means row order, dtypes, compression and metadata all landed identically — but it was demonstrated on a sample and is reported as such. The rebuild ran under Python 3.12.12, numpy 1.26.4, pandas 2.2.2.
Section 4
Container image
Status
Pending verification
The Docker configuration has been authored and statically validated but has not yet been executed in a real container runtime.
The configuration lives at research/Dockerfile and research/compose.yaml. It pins the base image by version, installs only from the committed lockfile after verifying the lockfile's digest, and sets the hash seed and thread counts that determinism depends on. None of that has been observed to work, because no image has been built here.
It is published in this state rather than withheld: an unbuilt Dockerfile that says so is useful to someone with a container runtime, and a green checkmark next to something nobody has executed would undermine the five metrics above it.