← polymarket-oracle
0/15 complete

Module 2: Market Research · 25 min

Using LLMs to Summarize Events Without Inventing Facts

// sabak

Turn this lesson into one checked practice output

By the end, you should be able to explain the core idea behind “Using LLMs to Summarize Events Without Inventing Facts” in your own words, apply it to one small real or sample task, and identify what still needs human review.

  1. 1

    Learn

    Read the 25-minute lesson without copying an output blindly.

  2. 2

    Try

    Use a small, non-sensitive example that you can inspect line by line.

  3. 3

    Review

    Check facts, fit, and risk; save one improvement note for next time.

Use an LLM to transform a controlled source packet, not to discover truth from memory. The model should extract claims, differences, timelines, and missing evidence with citations to provided source IDs. Human verification remains responsible for every fact that affects a forecast.

Prepare the Source Packet

Include only authorized material. Assign stable IDs such as S01, publisher, URL/document ID, publication/event/retrieval times, timezone, version, and exact relevant excerpt or structured fields. Remove secrets and personal/sensitive data. Preserve original files outside the model context.

Do not paste entire copyrighted articles when a short factual excerpt or official source is sufficient. Respect terms and quotation limits.

Use a Constrained Prompt

Task: build a verification table for the stated contract condition.
Use only sources S01-S08. For each claim return:
claim, supporting source IDs, contradicting source IDs,
event time, publication time, status (confirmed/unconfirmed/ambiguous),
and missing evidence.
If a fact is absent, write NOT IN PACK. Do not predict or advise action.

Request structured JSON or a table, then validate required fields. The model may still attach the wrong source, merge dates, or infer a fact. Open every cited source and compare.

Separate Fact, Interpretation, and Forecast

Use three layers:

  • Observed fact: directly supported by a source.
  • Interpretation: an explanation with assumptions.
  • Paper forecast: a probability recorded separately with method and timestamp.

Never let polished prose hide that only the third layer changed. If the source packet is incomplete, the correct output may be “insufficient evidence.”

Handle Breaking Updates

Append new sources; do not silently rewrite old packets. Re-run against a versioned packet and diff claims/status. Preserve the prior summary so hindsight cannot alter what was known. Flag retractions, corrections, draft/final transitions, and time-zone cutoffs.

Use deterministic code for timestamps, numeric aggregation, duplicate detection, and scoring. Use AI for semantic extraction and counterarguments, then verify.

Worked Example

Sources include a draft circular, a journalist’s report calling it final, and the agency’s later final circular. The LLM initially summarizes all three as “the rule was issued.” The reviewer corrects the timeline: draft at T1, report at T2, final at T3. If the market deadline was between T2 and T3, that difference controls the evaluation.

The correction is recorded as a model error category, and a future prompt requires document-status extraction.

Failure Cases

  • Asking the model “what happened?” without a packet.
  • Treating URLs in output as verified citations.
  • Combining draft, rumor, and final status.
  • Letting AI assign precise probability from vague evidence.
  • Uploading private messages or paid content without authorization.
  • Overwriting previous summaries after outcomes.
  • Sharing model-generated accusations about people.

🇵🇰 Pakistan Angle

OCR and translation can corrupt Urdu names, dates, numerals, and legal terms. Keep the original image/PDF, use human bilingual review, and cite page/section. Do not translate “notification,” “order,” “bill,” “act,” and “press release” as equivalent status.

For Pakistani financial/virtual-asset topics, use current PVARA, SECP, SBP, or other responsible sources and label the access date. Regulatory change makes model memory especially unsafe.

Hands-On Exercise

Build an eight-source synthetic or public packet. Prompt an LLM for the constrained verification table, validate its JSON/table, manually audit every claim, and produce an error log. Add one correction/update and show the version diff. End with evidence status only.

Completion Rubric

  • Complete: sources are controlled/versioned, output distinguishes status layers, every citation is manually verified, and errors feed the next rubric.
  • Needs revision: summary is useful but timestamps, contradiction, or missing-evidence behavior is inconsistent.
  • Not complete: model memory or generated citations are treated as truth or advice.

Sources

Key takeaway: Ground the model in a versioned packet, demand missing-evidence labels, and verify every claim outside the model.

Self-check

Before you mark Lesson 2.2 complete

  • Can I explain “Using LLMs to Summarize Events Without Inventing Facts” without reading the lesson back word for word?
  • Did I complete the lesson’s practice step on a real or clearly labelled sample task?
  • Did I check the result for invented facts, private data, unsafe actions, and mismatch with the brief?
← PreviousNext lesson →