This lab assembles scanning, immutable storage, normalization, filtering, and reporting into one reproducible pipeline. The deliverable is not “latest data.” It is a run bundle another reviewer can replay without network access and reconcile from raw bytes to eligibility counts.
Define a manifest before coding:
{
"run_id": "scan_20260719T080000Z_fixture_v3",
"app_version": "0.1.0",
"config_hash": "...",
"parser_version": "market_schema_v1",
"filter_version": "eligibility_v2",
"raw_hashes": [],
"started_at_utc": "...",
"completed_at_utc": null,
"status": "RUNNING"
}
The pipeline stages are explicit. capture writes raw pages. normalize validates records and writes normalized JSON Lines. filter appends eligibility decisions. summarize produces the denominator and quality metrics. seal checks declared artifacts, fills completion time, and writes a manifest hash. Each stage reads the prior stage’s immutable output; none reaches backward to edit it.
Use atomic publication. Write an artifact to a temporary file, flush and close it, calculate its hash, then rename it into the run directory. A crash must leave either the prior completed artifact or an obvious temporary file, never a half-valid object with a final name. The manifest remains RUNNING or FAILED until sealing succeeds.
Test reproducibility by running normalization and filtering twice from the same raw hashes and configuration. Outputs must match byte-for-byte after excluding generated wall-clock fields. If ordering can vary, sort deterministically by source ID and timestamp. Floating calculations should have documented precision.
Add an audit command that verifies hashes, expected schema versions, record counts between stages, and the equation eligible + rejected = normalized. It should report orphaned artifacts and an incomplete manifest. It must not repair automatically; repair can erase evidence.
Include a provenance query in the lab: given one dashboard row, print its normalized line, filter decision, raw page hash, capture metadata, parser version, and manifest. Time the replay and record peak disk use so the bundle has operational evidence as well as logical correctness. A reviewer should not need network access or undocumented local paths to complete the trace.
🇵🇰 Pakistan Angle
Package a small fixture so the complete lab runs offline on a modest laptop. A learner can capture new public data when connectivity permits, but grading uses the fixed bundle. This makes assessment fair across different internet and power conditions and prevents claims based on inaccessible live state.
Hands-On Exercise
Build the five commands, process at least two raw pages, and intentionally crash before sealing. Demonstrate audit detecting the incomplete run. Resume safely, seal the bundle, disconnect the network, and replay the report. Give the bundle to a reviewer with a one-page reproduction guide.
Completion Rubric
- Each stage consumes immutable prior-stage artifacts.
- Files publish atomically and hashes appear in the manifest.
- Counts reconcile and replay works without internet.
- Incomplete, corrupted, and orphaned states are detected.
- The bundle contains public/fixture data only and no execution surface.
Sources
Key takeaway: A versioned pipeline is finished only when its artifacts reconcile, seal, and replay from immutable evidence.