← Autonomous AI Agents
0/15 complete

Module 05 · Shipping a Real Agent System

Human-in-the-Loop Checkpoints for High-Stakes Actions

20 minfocused lesson5practical steps4grounded questions3source links
Open lesson + course map

On this lesson

Course outline

A human checkpoint is a decision interface with evidence and enforceable authority. It is not a vague notification or a button that approves unknown future actions. The reviewer needs competence, time, exact scope, alternatives, and a safe rejection path.

After this lesson, you can design a review packet and approval state machine for consequential effects.

// concept

Build the Review Packet

Show:

// prompt — copy me9 lines
requested action and reason
affected person/resource
exact before and after state
amount, recipient, audience, or permission change
source evidence and version
model uncertainty and evaluator failures
policy/check results
deadline and consequence of no action
approve, edit, reject, escalate

Do not bury risk in a full transcript. Highlight the facts required for the decision and link to authorized sources.

// concept

Use a Real Approval State Machine

// prompt — copy me3 lines
DRAFT → PENDING_REVIEW → APPROVED → EXECUTING → EXECUTED
                    ↘ REJECTED       ↘ FAILED/RECONCILE
                    ↘ EXPIRED

Approval contains reviewer identity, role, artifact hash, exact scope, timestamp, expiry, and reason. Execution revalidates authorization and that the artifact is unchanged. A rejection must stop downstream retries.

// concept

Prevent Automation Bias

Do not show the model recommendation as the only option. Provide source evidence and allow insufficient evidence. Sample approvals for quality, measure reversals and reviewer disagreement, and rotate or retrain reviewers when the queue creates rubber-stamping.

// worked_example

Worked Example

A Pakistani education business uses an agent to prepare scholarship applications. The agent extracts submitted facts and creates a completeness report. It cannot decide merit or reject a learner.

The reviewer sees published eligibility criteria, applicant-supplied evidence, missing items, and the exact proposed next message. They can request information, correct extraction, escalate, or approve the communication. Sensitive documents stay in the source system. Approval binds the message hash; edits require fresh review. Final selection remains with the authorized committee.

// failure_cases

Failure Cases to Diagnose

7 cases to diagnose

  • Reviewer sees only “AI confidence 92%”

    show evidence and criteria.

  • Approval survives content edit

    hash exact artifact and invalidate changes.

  • Notification is approval

    require authenticated action.

  • Queue deadline pressures automatic approval

    expire or escalate instead.

  • Reviewer lacks authority or expertise

    route by action type.

  • Reject triggers a repair loop forever

    stop or create a new bounded draft.

  • No audit of reviewer outcomes

    sample reversals and disagreements.

// pakistan_angle

Pakistan Angle

Use qualified humans for legal, tax, accounting, healthcare, education, hiring, lending, property, and financial actions as applicable. An AI-generated summary should not replace original Urdu/English documents, professional judgment, or the right of a person to correct data.

Design interfaces for practical connectivity: save drafts, allow resume, show PKT deadlines, and avoid approvals through insecure screenshots or personal chat. Never request an OTP as proof that a reviewer approved an agent action.

// hands_on

Hands-On Exercise

5 steps

  1. Choose one high-stakes action.

  2. Create the evidence-first review packet.

  3. Implement or diagram exact, expiring approval.

  4. Test edit-after-approval, rejection, expiry, unauthorized reviewer, and execution failure.

  5. Define a monthly review-quality audit.

// completion_rubric

Completion Rubric

6 checks — tick as you verify

0/6

// sources

Sources

3 official sources — check every claim yourself

// check_yourself

Check yourself

4 questions · answers and options are taken word-for-word from this course

0/4

  1. 1 / 4 · diagnose

    Your work shows this failure mode: “Reviewer sees only “AI confidence 92%”.” What does the lesson tell you to do about it?