The scanner is an anti-corruption layer between an external data source and your research code. Providers rename fields, return partial pages, throttle clients, and occasionally serve an error page with status 200. The wrapper absorbs those differences and emits either a validated raw snapshot or a typed failure. It never emits a plausible-looking empty dataset after an error.
Define one interface: scan(cursor: str | None) -> ScanPage. ScanPage contains records, next cursor, captured UTC time, source host, response hash, and schema observations. Only GET is allowed. Build URLs from a registered base and encoded parameters; do not accept arbitrary URLs from configuration.
Implement defenses in order: allowlist scheme and hostname, cap query length, connect/read timeouts, status check, response-size limit, content-type check, JSON parsing, top-level shape validation, and per-record minimum fields. A response larger than the documented cap should stop before full processing. Redirects must remain on the allowlist or fail.
Pagination needs its own guardrails. Track cursors seen in the current run, cap maximum pages and records, and stop on repeated cursors. Preserve each page before normalization. A partial scan is labelled partial with the final successful cursor and error category; it is not silently promoted to complete.
Use bounded retry with exponential delay and jitter for connection resets, timeouts, 429, and selected 5xx responses. Respect Retry-After when valid. Do not retry 400-series validation/access responses except 429. Log attempt counts and total elapsed time without response bodies that could contain unexpected sensitive material.
Contract test
Create a local fake HTTP server with fixtures for a valid page, HTML masquerading as success, a redirect to an unknown host, repeated cursor, oversized body, 429 then success, and malformed record. Assert the exact error type for each. Contract tests make undocumented provider changes visible before they corrupt research.
🇵🇰 Pakistan Angle
Optimize for interrupted networks: persist every verified page, resume only from a stored cursor, and keep the run visibly partial. Do not defeat provider limits with proxies, rotated identities, or parallel floods. A course learner in Pakistan uses public information within published access rules and does not treat endpoint reachability as trading permission.
Hands-On Exercise
Implement the interface against instructor fixtures and one documented public endpoint. Add the seven contract cases above. Generate scan_manifest.json with host, start/end, page hashes, cursor chain, total records, retry counts, completeness, and parser version. Review that no method other than GET appears.
Completion Rubric
- Host, redirect, size, time, page, and record limits fail closed.
- Pagination loops and partial scans are explicitly represented.
- Retry classification is bounded and observable.
- Raw pages are immutable and hashed before normalization.
- The adapter exposes no authentication or write operation.
Sources
Key takeaway: A defensive scanner converts unreliable external responses into explicit, bounded, auditable evidence.