An honest paper ledger preserves every eligible decision, abstention, rule violation, missing observation, and method change. It does not show only winning calls or use current information to rewrite old reasoning. The ledger evaluates forecasting discipline—not hypothetical wealth.
Use an Append-Only Schema
Record decision ID, market stable ID, contract/resolution snapshot hash, method version, evidence packet/cutoff, paper forecast, bid/ask/depth, hypothetical fill rule/result, fictional points, cluster, no-action gates, update/exit events, outcome/resolution source, score, errors, reviewer, and timestamps.
Separate:
- forecast quality (Brier/calibration);
- market comparison (forecast versus observed price metric at same time);
- paper execution estimate (spread/depth/fees assumptions);
- process quality (rule adherence, source verification, abstention);
- operational failures (missing/stale data, parser errors).
Do not merge them into “profit.”
Preserve an Audit Trail
Use immutable rows or versioned files. Corrections append a new event referencing the original and reason. Hash or version source packets/configuration where practical. Back up and restrict write access.
If a model or prompt changes, increment the method version and do not pool performance without labeling. Keep a change log and a future effective date to prevent opportunistic switching.
Review With Proper Denominators
Report the entire preselected universe, coverage, abstention, unresolved/invalid contracts, score distribution, calibration bins, worst errors, rule violations, and data failures. Show sample size and time period. A 70% hit rate can be meaningless if events had different base probabilities or the sample was selected afterward.
Use Brier score for frozen binary probabilities where appropriate, but interpret alongside calibration and sharpness/coverage. A method predicting 50% for everything may be calibrated in some samples yet uninformative.
Worked Example
A dashboard initially shows eight correct of ten paper calls. The audit reveals the watchlist began with 18 markets: four losing forecasts were excluded as “low confidence,” two unresolved were ignored, and two data failures were silently dropped. The corrected report shows all 18, including abstention and missingness.
The process lesson is more valuable than the flattering percentage.
Create a Review Cadence
Weekly: data quality, rule violations, source corrections, and emotional/process notes. Monthly or after sufficient resolutions: calibration, category slices, worst errors, and method change proposal. Lock changes until a future cohort; do not continually tune live paper records.
AI can cluster error notes but cannot decide whether its own reasoning was sound. Human reviewers open evidence.
Failure Cases
- Deleting losing or abstained decisions.
- Calling paper outcome realized return.
- Reconstructing entry prices after resolution.
- Ignoring spread, depth, fees, and missing data.
- Changing method without versioning.
- Comparing forecasts from different timestamps.
- Publishing performance from a tiny cherry-picked sample.
🇵🇰 Pakistan Angle
Keep the ledger free of real account/wallet/identity data and do not convert points to PKR. If sharing publicly, remove personal information and ensure claims cannot be mistaken for investment advice or verified income.
Use current official Pakistan regulator sources for risk statements and date the report. Regulatory changes should trigger scope review, not retroactive ledger edits.
Hands-On Exercise
Create a 25-row ledger from a preselected historical set or ongoing paper watchlist. Include abstentions and failures. Produce a one-page review with coverage, Brier/calibration where valid, paper-fill limitations, three worst errors, rule violations, and one future-version proposal.
Completion Rubric
- Complete: universe, append-only history, versions, denominators, calibration, execution limits, abstentions, and failures are visible.
- Needs revision: results are traceable but selection bias or method-change handling is incomplete.
- Not complete: paper results are sold as profit, or losing/missing records are removed.
Sources
- scikit-learn Brier score loss
- Polymarket official read-only CLOB client
- Polymarket official resolution subgraph
Key takeaway: Preserve every decision, abstention, failure, and rule version so the review measures process rather than storytelling.