Your first fetch should prove the read-only boundary and the failure behavior, not merely print JSON. A defensive client accepts a public URL from an allowlist, applies a short timeout, identifies itself, checks the response type, validates the shape, and saves the untouched payload with a timestamp. It never stores cookies, authorization headers, wallet material, or trade-capable SDK objects.
Create a tiny adapter whose only public method is fetch_markets. Keep the network layer separate from normalization so a saved fixture can exercise the parser without internet access.
from datetime import datetime, timezone
import hashlib, json, pathlib, requests
ALLOWED_HOST = "gamma-api.polymarket.com"
def fetch_public_markets(url: str, out_dir: str = "data/raw") -> pathlib.Path:
if not url.startswith(f"https://{ALLOWED_HOST}/"):
raise ValueError("host is not on the read-only allowlist")
response = requests.get(
url,
timeout=(3.05, 12),
headers={"User-Agent": "paper-research-course/1.0"},
)
response.raise_for_status()
if "application/json" not in response.headers.get("content-type", ""):
raise ValueError("expected JSON")
payload = response.json()
if not isinstance(payload, list):
raise ValueError("expected a list payload")
body = json.dumps(payload, sort_keys=True, ensure_ascii=False)
digest = hashlib.sha256(body.encode()).hexdigest()[:16]
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
target = pathlib.Path(out_dir) / f"{stamp}_{digest}.json"
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(body, encoding="utf-8")
return target
The connect/read timeout pair prevents a stalled connection from hanging forever. raise_for_status turns error responses into explicit failures. The content-type and top-level shape checks catch HTML error pages and undocumented schema changes. The content hash makes repeated payloads recognizable without deleting either observation.
Do not add automatic retries blindly. Retry only transient timeouts, connection failures, and selected server errors, with bounded exponential backoff and jitter. A validation error should fail immediately because repeating the same malformed response is not recovery. Honor documented rate limits and cache freshness. A research tool that hammers a free endpoint is not robust.
Failure drill
Test four cases using saved responses or mocks: success, timeout, HTTP 500, and JSON with the wrong top-level type. The expected outputs are one stored immutable file for success and one structured error event for each failure. The function must not silently return an empty list, because “no markets” and “fetch failed” mean different things.
🇵🇰 Pakistan Angle
Internet and power interruptions are normal engineering constraints for many Pakistani learners. A raw snapshot lets work resume after an outage without pretending the current API was available continuously. Keep the lab small enough to rerun on mobile data, but never bypass access controls or geographic rules. Public reachability is not permission for financial activity.
Hands-On Exercise
Implement the function, then replace the live URL with an instructor-approved fixture in your test. Record run_id, UTC start/end times, HTTP status, byte count, payload hash, and result. Prove that a wrong host and wrong schema both fail closed. Add a README line naming the endpoint as read-only market data.
Completion Rubric
- Only an allowlisted HTTPS public-data host is accepted.
- Timeouts, status checks, content type, and schema shape are enforced.
- Raw payloads are timestamped, hashed, and never overwritten.
- Tests distinguish empty data from failed retrieval.
- No authentication, order, wallet, or payment capability exists.
Sources
Key takeaway: A successful first fetch is a bounded, validated, reproducible evidence capture—not a one-line network request.