An AI research engine should make uncertainty easier to audit, not automate conviction. Use four tiers: deterministic extraction, optional model-assisted classification, independent verification, and human review. Every tier can abstain. A later tier may add evidence or reject an earlier result, but it cannot erase the record of what happened.
Tier 0 ingests only approved source documents and assigns immutable source IDs, capture times, hashes, and publication metadata. Tier 1 applies deterministic rules: date parsing, identifier matching, keyword extraction, and required-field checks. Tier 2 asks a model for structured candidate claims tied to source IDs. Tier 3 runs an independent check—another deterministic rule set or separately prompted model—against the same locked evidence. Tier 4 is the named human decision and abstention gate. The course calls this “four-tier research” after ingestion.
Define a case file with case_id, question version, evidence IDs, extraction version, prompt/model identifier when used, candidate claim, contradiction list, confidence category, and final status. Confidence is a review category such as LOW/MEDIUM/HIGH, not a calibrated probability unless you have separately validated it.
Models never browse freely inside the decision function. A retrieval step first builds an approved evidence packet. The model must cite only packet source IDs and return JSON matching a strict schema. Unsupported IDs, invented quotations, missing contradictions, or invalid JSON cause rejection. The system never converts prose directly into a paper action.
Use explicit statuses: INSUFFICIENT_EVIDENCE, CONFLICTING_EVIDENCE, READY_FOR_HUMAN_REVIEW, HUMAN_APPROVED_FOR_PAPER_TEST, and REJECTED. Approval means the hypothesis may enter a synthetic evaluation; it does not mean correct, profitable, legal, or suitable for real activity.
Define tier-level service objectives around integrity rather than speed: packet completeness, invalid source-ID rejection, abstention on missing rules, contradiction recall on a labelled fixture, and audit-record completeness. Measure model-free and model-assisted paths separately. If an optional model is unavailable, deterministic extraction and human review continue; the system must not invent a summary or reuse a result from a different packet hash.
🇵🇰 Pakistan Angle
Local reporting may mix English, Urdu, and Roman Urdu and may lack machine-readable archives. Preserve original text and translation separately, record the translator/model, and ask a bilingual reviewer for ambiguous claims. Never upload private WhatsApp messages, CNICs, or personal records as evidence.
Hands-On Exercise
Create one case from three supplied public documents, including one contradiction. Run deterministic extraction, produce model-shaped JSON manually or with an optional provider, validate source IDs, and complete a human review form. Then remove a critical source and prove the status becomes INSUFFICIENT_EVIDENCE.
Completion Rubric
- Evidence ingestion and four review tiers have separate records.
- Every candidate claim points to allowed immutable source IDs.
- Contradictions and abstention statuses are first-class fields.
- Model output cannot directly create a paper decision.
- Human approval is scoped only to paper evaluation.
Sources
Key takeaway: Layered research earns trust by preserving evidence, disagreement, abstention, and human accountability.