Module 04 · Paper Decision Logic
Logging Simulated Decisions for Honest Review
Open lesson + course map
On this lesson
Course outline
Module 1 · Prediction Markets 101
Module 2 · Market Research
Module 3 · Signal Aggregation
Module 4 · Paper Decision Logic
Module 5 · Risk and Responsible Research
An honest paper ledger preserves every eligible decision, abstention, rule violation, missing observation, and method change. It does not show only winning calls or use current information to rewrite old reasoning. The ledger evaluates forecasting discipline—not hypothetical wealth.
// concept
Use an Append-Only Schema
Record decision ID, market stable ID, contract/resolution snapshot hash, method version, evidence packet/cutoff, paper forecast, bid/ask/depth, hypothetical fill rule/result, fictional points, cluster, no-action gates, update/exit events, outcome/resolution source, score, errors, reviewer, and timestamps.
Separate:
- forecast quality (Brier/calibration);
- market comparison (forecast versus observed price metric at same time);
- paper execution estimate (spread/depth/fees assumptions);
- process quality (rule adherence, source verification, abstention);
- operational failures (missing/stale data, parser errors).
Do not merge them into “profit.”
// concept
Preserve an Audit Trail
Use immutable rows or versioned files. Corrections append a new event referencing the original and reason. Hash or version source packets/configuration where practical. Back up and restrict write access.
If a model or prompt changes, increment the method version and do not pool performance without labeling. Keep a change log and a future effective date to prevent opportunistic switching.
// concept
Review With Proper Denominators
Report the entire preselected universe, coverage, abstention, unresolved/invalid contracts, score distribution, calibration bins, worst errors, rule violations, and data failures. Show sample size and time period. A 70% hit rate can be meaningless if events had different base probabilities or the sample was selected afterward.
Use Brier score for frozen binary probabilities where appropriate, but interpret alongside calibration and sharpness/coverage. A method predicting 50% for everything may be calibrated in some samples yet uninformative.
// worked_example
Worked Example
A dashboard initially shows eight correct of ten paper calls. The audit reveals the watchlist began with 18 markets: four losing forecasts were excluded as “low confidence,” two unresolved were ignored, and two data failures were silently dropped. The corrected report shows all 18, including abstention and missingness.
The process lesson is more valuable than the flattering percentage.
// concept
Create a Review Cadence
Weekly: data quality, rule violations, source corrections, and emotional/process notes. Monthly or after sufficient resolutions: calibration, category slices, worst errors, and method change proposal. Lock changes until a future cohort; do not continually tune live paper records.
AI can cluster error notes but cannot decide whether its own reasoning was sound. Human reviewers open evidence.
// failure_cases
Failure Cases
7 cases to diagnose
Deleting losing or abstained decisions.
Calling paper outcome realized return.
Reconstructing entry prices after resolution.
Ignoring spread, depth, fees, and missing data.
Changing method without versioning.
Comparing forecasts from different timestamps.
Publishing performance from a tiny cherry-picked sample.
// pakistan_angle
Pakistan Angle
Keep the ledger free of real account/wallet/identity data and do not convert points to PKR. If sharing publicly, remove personal information and ensure claims cannot be mistaken for investment advice or verified income.
Use current official Pakistan regulator sources for risk statements and date the report. Regulatory changes should trigger scope review, not retroactive ledger edits.
// hands_on
Hands-On Exercise
Create a 25-row ledger from a preselected historical set or ongoing paper watchlist. Include abstentions and failures. Produce a one-page review with coverage, Brier/calibration where valid, paper-fill limitations, three worst errors, rule violations, and one future-version proposal.
// completion_rubric
Completion Rubric
3 grading bands
- Complete
universe, append-only history, versions, denominators, calibration, execution limits, abstentions, and failures are visible.
- Needs revision
results are traceable but selection bias or method-change handling is incomplete.
- Not complete
paper results are sold as profit, or losing/missing records are removed.
// sources
Sources
3 official sources — check every claim yourself