Module 03 · Signal Aggregation
Designing a Multi-Source Signal Pipeline
Open lesson + course map
On this lesson
Course outline
Module 1 · Prediction Markets 101
Module 2 · Market Research
Module 3 · Signal Aggregation
Module 4 · Paper Decision Logic
Module 5 · Risk and Responsible Research
A multi-source pipeline should preserve what each source said, when it said it, and how it relates to the contract. It should not collapse repeated reporting into fake independent confirmation or convert weak text into a confident number.
// concept
Separate Pipeline Layers
Use immutable stages:
- Ingest: collect authorized official/public records and read-only market observations.
- Raw store: preserve original payload/content hash, source URL/ID, retrieval time, and terms-compliant metadata.
- Normalize: parse dates/timezones, entities, document status, units, and stable identifiers.
- Deduplicate/lineage: link copied reports to a common origin and version chain.
- Extract: identify claims, support/contradiction, and relevance to resolution.
- Review: human approval for decisive claims and ambiguous status.
- Aggregate: compute transparent summaries from approved structured rows.
- Publish: output a paper-research brief with source IDs, uncertainty, and freshness.
Never let an LLM write directly into the approved evidence table without validation.
// concept
Define a Claim Schema
Each claim row needs claim ID, market ID, source ID, original/secondary flag, event time, publication time, retrieval time, timezone, document version, stance (supports/contradicts/neutral), relevance, status (confirmed/unconfirmed/ambiguous), reviewer, and notes.
Keep source quality dimensions separate. “Official” does not guarantee timeliness; “fast” does not guarantee independence. A later correction supersedes a claim but does not erase the historical record.
// concept
Handle Independence and Duplication
Create a lineage graph: report B cites report A, which cites one anonymous post. This is one origin, not three signals. Cluster verbatim/near-verbatim content as a review aid, then verify manually. Avoid accusing publishers of copying based only on similarity.
For independent evidence, ask whether sources observed the fact separately, used different primary records, or merely referenced the same announcement.
// concept
Build for Failure
Use timeouts, bounded retries/backoff, rate-limit compliance, schema validation, dead-letter review, idempotent ingestion, and freshness alerts. A missing feed must be marked missing; it cannot become a neutral or negative signal. Store parser/runtime versions and tests for sample payloads.
Do not scrape access-controlled or personal data. Minimize stored content and define retention/deletion.
// worked_example
Worked Example
Five headlines report that an agency approved a proposal. Lineage shows all five rely on one wire report, while the official docket says “accepted for review.” The pipeline keeps five documents but one evidence origin and status unconfirmed. When the final order appears, it creates a new primary claim and preserves the prior timeline.
// failure_cases
Failure Cases
7 cases to diagnose
Counting articles rather than independent evidence.
Overwriting corrected or superseded documents.
Mixing event, publication, and retrieval times.
Treating feed failure as no event.
Letting AI create source IDs or quotes.
Hiding parser errors behind a dashboard score.
Collecting more personal/full-text data than needed.
// pakistan_angle
Pakistan Angle
Official Pakistani sites may publish PDFs, scans, English/Urdu versions, or delayed updates. Preserve original files/hashes where lawful, perform OCR as a derived layer, and require human review for names, dates, and legal status. Do not bypass site protections.
Plan for connectivity outages with idempotent resume and stale markers. Store UTC plus original timezone and display PKT explicitly.
// hands_on
Hands-On Exercise
Design the eight-stage pipeline for one market using ten synthetic/public sources. Produce data dictionary, lineage graph, three parser tests, retry/staleness rules, a five-row approved claim table, and a failure drill. No live-trading fields or credentials may exist.
// completion_rubric
Completion Rubric
3 grading bands
- Complete
raw/normalized/approved layers, lineage, time semantics, failures, privacy, and reviewer ownership are explicit.
- Needs revision
sources are combined but independence, corrections, or stale-feed behavior remains weak.
- Not complete
article count becomes confidence or AI output bypasses evidence review.
// sources
Sources
3 official sources — check every claim yourself