Module 01 · Market Systems and Safety — Pehle Boundaries Samjho
Dataset Selection — Liquidity, Bias, and Missing Data
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
A dataset is a set of inclusion decisions. Selecting only popular, resolved, clean markets can make a weak idea look excellent because difficult cases disappeared before evaluation. Professional research writes the sampling frame first, then applies it mechanically and reports every exclusion.
Begin with the target question: “How well do timestamped public probabilities align with later binary outcomes under a paper-only evaluation?” This is narrower than “Can we predict markets?” Define a fixed observation window, resolution window, allowed market states, minimum snapshot frequency, and fields required for analysis. Freeze these rules before seeing the score.
Liquidity is not a quality guarantee. Thin data often has wide or absent quotes; high activity can still be concentrated in a few topics. Treat spread, depth proxy, update count, and observation coverage as quality features. Never backfill a missing quote using the final outcome or a later value. If a midpoint requires both bid and ask, mark it unavailable whenever either side is missing.
Build an exclusion table:
| Reason | Count | Rule | Bias risk |
|---|---|---|---|
| No timestamped quote | measured | exclude from price evaluation | removes poorly covered markets |
| Rule changed | measured | analyze as separate version | revised markets may be harder |
| Unresolved by cutoff | measured | keep in denominator, label pending | resolved-only samples look cleaner |
| Duplicate identifier | measured | retain first raw record, quarantine set | duplicates distort weighting |
The word “measured” matters: fill counts from code, not memory. Publish the full denominator: discovered, fetched, validated, eligible, excluded by reason, pending, and evaluated. A chart based on 80 rows is misleading if 920 discovered rows vanished without explanation.
Bias checklist
Survivorship bias appears when deleted or unresolved markets vanish. Lookahead bias appears when later facts influence an earlier feature. Selection bias appears when thresholds are tuned after seeing results. Topic imbalance appears when one repeated theme dominates. Schema drift appears when an API field changes meaning across dates. Control these with raw snapshots, time-based splits, predeclared rules, grouped reports, and versioned parsers.
// pakistan_angle
Pakistan Angle
Local political, economic, and sports questions may have thinner or less consistent public data than heavily traded international topics. Do not claim a model understands Pakistan from a handful of visible examples. Report country/topic coverage and language limitations. Do not scrape personal data, closed groups, or restricted sources to “improve” coverage.
// hands_on
Hands-On Exercise
Take two saved fixture files. Write dataset_card.md with purpose, time range, source, fields, inclusion rules, exclusions, known gaps, and prohibited uses. Generate the denominator table in code. Then create one adversarial check: include unresolved rows in the discovered count and verify they cannot silently disappear from the report.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself