Random train/test splits leak time structure. Walk-forward testing trains or tunes on an earlier window, freezes the pipeline, evaluates on the next window, then advances. It approximates how evidence would have arrived without implying that paper results justify later real use.
Define folds by time before seeing metrics. For each fold, fit preprocessors, thresholds, and optional models only on training data. Validation may choose among preregistered alternatives; the test window is touched once. Resolution labels and revised documents must never enter feature construction before their availability times.
Track market or event families so near-duplicates do not straddle train and test. Fit language vocabularies, imputers, and calibration transforms inside each fold. Record code commit, configuration, data hashes, fold boundaries, and feature schema.
Report fold-level and aggregate metrics: Brier score, calibration bins, paper coverage, abstention reasons, synthetic drawdown in points, data-quality failures, and assumed cost sensitivity. Show dispersion; an average can hide one catastrophic period. Compare with frozen simple baselines.
After the final test, do not keep tuning against it. New ideas require a new future paper window. Multiple hypothesis attempts should be counted and disclosed because selecting the best of many tests inflates apparent evidence.
Create a fold manifest before each run and a sealed result afterward. The manifest lists train, validation, and test boundaries; event-family exclusions; preprocessing versions; label availability cutoff; baseline; primary metric; and stress grid. The result lists every attempted configuration, including failures. Compare expanding-window and fixed-window variants only if both were preregistered, because choosing the better one afterward is another hidden hypothesis test.
Include a deliberately boring operational metric: percentage of test cases that could be reproduced byte-for-byte from stored artifacts. A high score without reproducibility is not acceptable. If a source was revised or lost, mark the fold non-reproducible and exclude it from any headline comparison while preserving the original report.
🇵🇰 Pakistan Angle
If Pakistan-topic cases appear only in one period, report that distribution rather than claiming geographic generalization. Use the exercise as portfolio proof of evaluation engineering. No score converts to a promise, signal, or invitation to connect funds.
Hands-On Exercise
Create three chronological folds from synthetic/versioned data. Add a deliberately leaky feature derived from final resolution, prove it looks unusually strong, then remove it and document the difference. Generate a model card listing fold boundaries, attempted hypotheses, limitations, and paper-only use.
Completion Rubric
- Folds and alternatives are defined before test evaluation.
- Preprocessing fits inside each training window.
- Event families and availability times prevent leakage.
- Fold dispersion, abstention, costs, and attempt counts are reported.
- The conclusion makes no real-later or profit claim.
Sources
Key takeaway: Walk-forward tests preserve chronology and expose instability; they do not create permission for real execution.