A research hypothesis predicts a measurable association; it does not invent a cause. “Forecast error differs by time remaining to resolution” can be tested. “Prices become accurate because smart traders arrive late” is a causal story requiring different evidence. Keep the testable statement, mechanism speculation, and result separate.
Write a preregistration with population, observation unit, timestamp rule, horizon buckets, outcome label, primary metric, exclusions, subgroup checks, and stopping rule. Freeze it before looking at evaluation results. Example horizon buckets might be greater than 30 days, 8–30 days, 2–7 days, and under 48 hours, chosen for interpretability rather than optimized performance.
Use one observation per market per scheduled horizon to prevent frequently sampled markets from dominating. Ensure each observation uses information available at that time. The final label is joined only during evaluation. Pending markets stay in the denominator report but cannot receive labels.
Choose Brier score for probability forecasts and report calibration tables. Compare with a simple baseline such as the overall historical base rate estimated only from training periods. Include sample count, topic mix, missingness, spread distribution, and uncertainty intervals by bucket. A lower score in one period is an observation, not proof it will recur.
Potential confounders include topic, liquidity proxy, rule clarity, market age, event news intensity, and dataset coverage. Report stratified results where sample size permits. Do not control variables after seeing which adjustment makes the hypothesis look best.
Predefine a minimum reporting threshold for each bucket and combine or suppress unstable slices according to that rule, never according to whether their result looks favorable. Show both market count and independent event-family count. Run a negative-control feature that should have no predictive timing relationship; suspicious performance can reveal leakage, duplicated cases, or a broken cutoff join.
🇵🇰 Pakistan Angle
Pakistan-related samples may be small and concentrated around a few events. Do not generalize from them to “Pakistani markets” or public understanding. Label geographic/topic coverage and retain an “insufficient sample” outcome. This remains paper research, not a local trading recommendation.
Hands-On Exercise
Draft a one-page preregistration and generate a synthetic dataset with known calibration differences across horizons. Run the frozen analysis, then add a hidden confounder to show how an apparent horizon pattern can change after stratification. Record both results without rewriting the original hypothesis.
Completion Rubric
- The hypothesis is measurable and avoids causal wording.
- Horizon buckets, metrics, baselines, and exclusions are frozen.
- Features respect observation-time cutoffs.
- Results include counts, uncertainty, missingness, and subgroup limits.
- Findings are not converted into profit or live-use claims.
Sources
- scikit-learn Brier score documentation
- scikit-learn calibration guide
- NIST AI RMF measurement playbook
Key takeaway: A good hypothesis fixes a measurable association in advance, preserves every reported denominator transparently, and leaves unsupported causal stories explicitly unclaimed.