Module 04 · AI Research Engine — Extraction Se Human Review Tak
Tier 2 — Independent Model Check With Source IDs
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
A model check is useful for comparing language and surfacing contradictions, but its output is untrusted. Give it a frozen evidence packet, a narrow task, allowed source IDs, a JSON schema, and an instruction to abstain. Do not ask for market advice or a predicted winner.
The task can be: “Extract claims relevant to the stated resolution rule, list supporting and contradicting source IDs, and state whether evidence is sufficient for human review.” The packet includes only documents available by the observation cutoff. Each excerpt carries source_id, title, captured time, publication time if known, and text. Randomly generated IDs prevent the model from inferring authority from names.
Require output fields: case_id, claims[], supporting_source_ids[], contradicting_source_ids[], missing_information[], status, and reason. Reject an ID not present in the packet. Reject quotations that do not match source spans. Reject READY_FOR_REVIEW when required contradictions or missing-information fields are absent.
Independence means the check does not see a desired paper direction or previous model’s conclusion. It may see deterministic extracts, but not a label like “bullish.” If two model checks are used, vary prompts only through a versioned experiment and do not cherry-pick the preferred answer. Disagreement becomes a review feature.
Log provider, model identifier, prompt hash, schema version, evidence-packet hash, response hash, latency, and token/cost measurements when available. Volatile pricing and model behavior must be checked from current official provider information; do not hardcode a permanent ranking.
Build an evaluation set where the correct behavior includes both extraction and abstention. Score valid source-ID use, unsupported-claim rate, contradiction capture, exact-span accuracy, schema validity, and stability across repeated runs. Never grade only agreement with a human direction; that rewards confident copying. Review failures by source type and language, then decide whether the model path should remain enabled for that slice.
// pakistan_angle
Pakistan Angle
For bilingual sources, provide original and clearly labelled translation. Ask the model to cite the original source ID and flag uncertain translation. Keep public-document excerpts minimal to control data use and cost. Never send private community messages or identity documents.
// hands_on
Hands-On Exercise
Construct two evidence packets: one sufficient and one missing the key rule. Run or manually simulate the same schema-constrained prompt. Write validators for allowed IDs, exact quotation spans, required fields, and status logic. Demonstrate the incomplete packet abstains.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself