Module 03 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Market Filters — Time, Status, Spread, and Coverage
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
Filters define which records enter a research sample. If thresholds change after seeing results, the backtest becomes a story fitted to the answer. Write a filter specification first, version it, and make the pipeline produce both accepted and rejected records with reason codes.
Start with four independent dimensions. Time checks observation time, event cutoff, and allowed horizon. Status distinguishes open, closed, resolved, cancelled, and unknown. Spread checks whether both quote sides exist and whether their difference is within a predeclared descriptive threshold. Coverage measures how many expected observation windows were actually captured.
Filters must not overwrite. Produce a decision object:
{
"source_id": "fixture-27",
"filter_version": "eligibility_v2",
"eligible": false,
"reasons": ["STALE_QUOTE", "LOW_COVERAGE"],
"observed_at_utc": "2026-07-19T08:00:00Z",
"input_hash": "..."
}Evaluate all reasons rather than stopping at the first. This supports a full exclusion matrix and reveals interactions. Keep unknown status as unknown; do not map it to open. Treat boundary times consistently, preferably half-open intervals such as [start, end). Test exactly-on-boundary records.
Spread is a data-quality and paper-cost input, not a signal. A 0.02 quoted spread does not guarantee a fill cost of 0.02. Coverage is also not volume: ten snapshots captured in one minute do not cover a ten-hour window. Define expected windows and count distinct covered buckets.
Report filters as a waterfall: discovered, schema-valid, within time, known status, quote-complete, coverage-eligible, final sample. Include multi-reason counts separately so readers do not mistake waterfall order for the only cause of exclusion.
Add a reconciliation assertion after every stage. If 250 normalized rows produce 180 eligible and 68 rejected, the pipeline must stop because two rows are unexplained. Store the exact filter configuration and code version beside the counts so a later threshold change cannot inherit the old report. Review a random accepted and rejected sample manually; this catches inverted comparisons and timezone mistakes that totals alone may miss.
// pakistan_angle
Pakistan Angle
Internet outages can reduce observation coverage systematically around local times. Report coverage by UTC and Pakistan-time hour to detect this operational bias. Do not “repair” gaps using later quotes. If a fixture covers mainly foreign topics, state that it does not represent Pakistani events.
// hands_on
Hands-On Exercise
Write a versioned filter function and twelve boundary fixtures: exact cutoff, one second before/after, unknown status, missing bid, missing ask, negative spread, very wide spread, duplicate timestamps, sparse windows, dense single window, and adequate coverage. Produce accepted and rejected CSVs plus a reason-count report.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself