Module 03 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Scanner Module — Public API Ka Defensive Wrapper
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
The scanner is an anti-corruption layer between an external data source and your research code. Providers rename fields, return partial pages, throttle clients, and occasionally serve an error page with status 200. The wrapper absorbs those differences and emits either a validated raw snapshot or a typed failure. It never emits a plausible-looking empty dataset after an error.
Define one interface: scan(cursor: str | None) -> ScanPage. ScanPage contains records, next cursor, captured UTC time, source host, response hash, and schema observations. Only GET is allowed. Build URLs from a registered base and encoded parameters; do not accept arbitrary URLs from configuration.
Implement defenses in order: allowlist scheme and hostname, cap query length, connect/read timeouts, status check, response-size limit, content-type check, JSON parsing, top-level shape validation, and per-record minimum fields. A response larger than the documented cap should stop before full processing. Redirects must remain on the allowlist or fail.
Pagination needs its own guardrails. Track cursors seen in the current run, cap maximum pages and records, and stop on repeated cursors. Preserve each page before normalization. A partial scan is labelled partial with the final successful cursor and error category; it is not silently promoted to complete.
Use bounded retry with exponential delay and jitter for connection resets, timeouts, 429, and selected 5xx responses. Respect Retry-After when valid. Do not retry 400-series validation/access responses except 429. Log attempt counts and total elapsed time without response bodies that could contain unexpected sensitive material.
Contract test
Create a local fake HTTP server with fixtures for a valid page, HTML masquerading as success, a redirect to an unknown host, repeated cursor, oversized body, 429 then success, and malformed record. Assert the exact error type for each. Contract tests make undocumented provider changes visible before they corrupt research.
// pakistan_angle
Pakistan Angle
Optimize for interrupted networks: persist every verified page, resume only from a stored cursor, and keep the run visibly partial. Do not defeat provider limits with proxies, rotated identities, or parallel floods. A course learner in Pakistan uses public information within published access rules and does not treat endpoint reachability as trading permission.
// hands_on
Hands-On Exercise
Implement the interface against instructor fixtures and one documented public endpoint. Add the seven contract cases above. Generate scan_manifest.json with host, start/end, page hashes, cursor chain, total records, retry counts, completeness, and parser version. Review that no method other than GET appears.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself