A dataset is a set of inclusion decisions. Selecting only popular, resolved, clean markets can make a weak idea look excellent because difficult cases disappeared before evaluation. Professional research writes the sampling frame first, then applies it mechanically and reports every exclusion.
Begin with the target question: “How well do timestamped public probabilities align with later binary outcomes under a paper-only evaluation?” This is narrower than “Can we predict markets?” Define a fixed observation window, resolution window, allowed market states, minimum snapshot frequency, and fields required for analysis. Freeze these rules before seeing the score.
Liquidity is not a quality guarantee. Thin data often has wide or absent quotes; high activity can still be concentrated in a few topics. Treat spread, depth proxy, update count, and observation coverage as quality features. Never backfill a missing quote using the final outcome or a later value. If a midpoint requires both bid and ask, mark it unavailable whenever either side is missing.
Build an exclusion table:
| Reason | Count | Rule | Bias risk |
|---|---|---|---|
| No timestamped quote | measured | exclude from price evaluation | removes poorly covered markets |
| Rule changed | measured | analyze as separate version | revised markets may be harder |
| Unresolved by cutoff | measured | keep in denominator, label pending | resolved-only samples look cleaner |
| Duplicate identifier | measured | retain first raw record, quarantine set | duplicates distort weighting |
The word “measured” matters: fill counts from code, not memory. Publish the full denominator: discovered, fetched, validated, eligible, excluded by reason, pending, and evaluated. A chart based on 80 rows is misleading if 920 discovered rows vanished without explanation.
Bias checklist
Survivorship bias appears when deleted or unresolved markets vanish. Lookahead bias appears when later facts influence an earlier feature. Selection bias appears when thresholds are tuned after seeing results. Topic imbalance appears when one repeated theme dominates. Schema drift appears when an API field changes meaning across dates. Control these with raw snapshots, time-based splits, predeclared rules, grouped reports, and versioned parsers.
🇵🇰 Pakistan Angle
Local political, economic, and sports questions may have thinner or less consistent public data than heavily traded international topics. Do not claim a model understands Pakistan from a handful of visible examples. Report country/topic coverage and language limitations. Do not scrape personal data, closed groups, or restricted sources to “improve” coverage.
Hands-On Exercise
Take two saved fixture files. Write dataset_card.md with purpose, time range, source, fields, inclusion rules, exclusions, known gaps, and prohibited uses. Generate the denominator table in code. Then create one adversarial check: include unresolved rows in the discovered count and verify they cannot silently disappear from the report.
Completion Rubric
- The research question and sampling frame are fixed before scoring.
- Every exclusion has a mechanical rule and measured count.
- Missing bid/ask data is not imputed from future information.
- Survivorship, lookahead, selection, topic, and schema risks are documented.
- The dataset card states paper-research use only.
Sources
- scikit-learn time-series cross-validation
- NIST AI RMF Playbook
- Polymarket resolution subgraph repository
Key takeaway: Honest dataset selection preserves the denominator and exposes who was excluded, when, and why.