Signal weights should come from predeclared, reviewable evidence—not how persuasive a source sounds today. Reliability is multi-dimensional and topic-specific. A weighted score remains a model output with uncertainty; it is not a probability or instruction unless calibrated and validated for that exact use.
Score Dimensions Separately
For each source/claim, evaluate:
- authority/proximity: named resolution source, primary observer, secondary report;
- independence: original evidence versus common upstream source;
- specificity: directly addresses the contract condition or only background;
- timeliness: event/publication time relative to deadline;
- status clarity: final, provisional, draft, rumor, correction;
- historical calibration: prior comparable claims resolved correctly under a defined sample;
- transparency: method, document, data, and correction path;
- conflict/incentive: disclosed interest or influence over outcome.
Keep these columns visible. Collapsing them into one number can hide why evidence matters.
Start With Rules, Then Learn Weights
Use hard gates first: exclude fabricated/unverifiable sources, duplicate origins, out-of-scope evidence, nonpublic/illegally obtained information, and claims that do not address resolution. The named resolution source may determine settlement even if another source better describes reality.
For exploratory aggregation, assign simple prewritten weights and run sensitivity analysis. Change each plausible weight across a range and see whether the conclusion flips. If it does, label the result unstable.
Historical learning requires an unbiased dataset, frozen outcomes, leakage controls, and time-valid features. Do not use information published after the forecast timestamp. Split evaluation by time and event category.
Avoid Double Counting
Group dependent sources and allocate weight to the underlying origin rather than every repetition. Correlated indicators—three polls sharing sampling problems, or many articles quoting one release—do not provide three independent confirmations.
An LLM can suggest dependency links, but a human must verify citations and reasoning. Do not allow it to score its own generated summary as a source.
Worked Example
An official draft, two reports citing the draft, one anonymous claim, and a final order exist. Before the final order, the pipeline weights the draft as evidence of intent but not completion; copied reports add context but no independent confirmation. After the final order, resolution relevance changes.
A sensitivity table shows the paper view is stable only after the final document. The report states this instead of claiming the model “predicted the news.”
Evaluate Calibration
For resolved binary forecasts, record Brier score (forecast - outcome)^2 and calibration bins over many preselected events. Also track coverage, abstention, ambiguity, and worst errors. A low score on a tiny or cherry-picked set is weak evidence.
Never tune weights on the same outcomes used to advertise performance.
Failure Cases
- Giving fixed trust scores to entire organizations across topics.
- Counting copied sources separately.
- Using post-event information in features.
- Converting a weighted sum directly into precise probability.
- Optimizing and testing on the same markets.
- Ignoring resolution ambiguity or invalid markets.
- Calling backtest calibration a profit guarantee.
🇵🇰 Pakistan Angle
Different Pakistani agencies have authority over different facts. The ECP may be decisive for an election record; PVARA for virtual-asset service regulation; SBP for its circulars. Do not create a universal “government source” weight.
English/Urdu duplicated releases are one origin. Translation differences need review, not double weight. Preserve notification numbers and dates.
Hands-On Exercise
Take 20 resolved public events selected by a prewritten rule. Create dimension scores, origin clusters, predeclared weights, and paper forecasts using only pre-cutoff evidence. Calculate Brier score/calibration bins, run sensitivity analysis, and document two worst errors.
Completion Rubric
- Complete: weights are gated, source-specific, time-valid, dependency-aware, sensitivity-tested, and evaluated out of sample where possible.
- Needs revision: scoring is transparent but calibration sample or independence review is weak.
- Not complete: persuasive language or article count becomes precise confidence or promised returns.
Sources
- scikit-learn Brier score loss
- NIST AI Risk Management Framework
- Polymarket official resolution subgraph
Key takeaway: Weight evidence by transparent, time-valid dimensions and dependencies, then test calibration and sensitivity rather than manufacturing confidence.