Hands-On Lab: Versioned Data Pipeline Build

This lab assembles scanning, immutable storage, normalization, filtering, and reporting into one reproducible pipeline. The deliverable is not “latest data.” It is a run bundle another reviewer can replay without network access and reconcile from raw bytes to eligibility counts.

Define a manifest before coding:

{
  "run_id": "scan_20260719T080000Z_fixture_v3",
  "app_version": "0.1.0",
  "config_hash": "...",
  "parser_version": "market_schema_v1",
  "filter_version": "eligibility_v2",
  "raw_hashes": [],
  "started_at_utc": "...",
  "completed_at_utc": null,
  "status": "RUNNING"
}

The pipeline stages are explicit. capture writes raw pages. normalize validates records and writes normalized JSON Lines. filter appends eligibility decisions. summarize produces the denominator and quality metrics. seal checks declared artifacts, fills completion time, and writes a manifest hash. Each stage reads the prior stage’s immutable output; none reaches backward to edit it.

Use atomic publication. Write an artifact to a temporary file, flush and close it, calculate its hash, then rename it into the run directory. A crash must leave either the prior completed artifact or an obvious temporary file, never a half-valid object with a final name. The manifest remains RUNNING or FAILED until sealing succeeds.

Test reproducibility by running normalization and filtering twice from the same raw hashes and configuration. Outputs must match byte-for-byte after excluding generated wall-clock fields. If ordering can vary, sort deterministically by source ID and timestamp. Floating calculations should have documented precision.

Add an audit command that verifies hashes, expected schema versions, record counts between stages, and the equation eligible + rejected = normalized. It should report orphaned artifacts and an incomplete manifest. It must not repair automatically; repair can erase evidence.

Include a provenance query in the lab: given one dashboard row, print its normalized line, filter decision, raw page hash, capture metadata, parser version, and manifest. Time the replay and record peak disk use so the bundle has operational evidence as well as logical correctness. A reviewer should not need network access or undocumented local paths to complete the trace.