Applied AI Builder
0/18 complete

Module 03 · Data and Multimodal Work

Lab: Produce an Evidence-Backed Decision Memo

45 minfocused lesson8practical steps4grounded questions5source links
Open lesson + course map

On this lesson

Course outline

An evidence-backed decision memo states a decision, separates verified facts from calculations and assumptions, shows how each material claim maps to a source, tests whether the recommendation changes under reasonable uncertainty, and names a human owner. AI may organize and challenge the memo; it must not invent missing evidence or make the accountable decision. The deliverable is not “a good answer.” It is a reviewable chain from source to calculation to recommendation.

This capstone combines tabular cleaning and multimodal provenance. Use only synthetic evidence. The scenario below is fictional so no vendor, rate, or market claim should be treated as current.

// concept

Frame a Decision That Evidence Can Answer

Weak question: “Which delivery partner is best?”

Testable question: “For our synthetic 90-day Lahore pilot, which option should receive a four-week controlled test, given approved priorities of reliability, cost per completed delivery, and support responsiveness?”

Write the decision owner, deadline, options, constraints, and reversible next step. A pilot recommendation is different from a permanent contract. Define the comparison period and population. If one option handled different routes or order sizes, a raw average may be unfair.

Create an evidence register:

Evidence IDSourceScope/dateOwnerMaterial useLimitation
E001synthetic deliveries.csv90-day lab fixturelearnercompletion and cost metricsnot real operations
E002synthetic quotation PDFlab version 1learnerterms and exclusionsunsigned fixture
E003synthetic interview audio35 secondslearnersupport concernsone fictional speaker

Hash every file and preserve raw and derived versions. Every memo claim must cite an evidence ID plus row/filter, page/section, or timestamp.

// worked_example

Worked Example: Calculate Before Asking for Narrative

Create options.csv:

// csv3 lines
option,completed_deliveries,attempted_deliveries,total_cost_pkr,median_support_hours,terms_verified
Option A,88,100,30800,5,true
Option B,81,90,27000,2,false

These figures are invented only to demonstrate calculations. Create score.py:

// python57 lines
from pathlib import Path
import json
import pandas as pd

df = pd.read_csv("options.csv")
required = {
    "option", "completed_deliveries", "attempted_deliveries",
    "total_cost_pkr", "median_support_hours", "terms_verified"
}
missing = required - set(df.columns)
if missing:
    raise ValueError(f"Missing columns: {sorted(missing)}")

if (df["attempted_deliveries"] <= 0).any():
    raise ValueError("attempted_deliveries must be positive")
if (df["completed_deliveries"] <= 0).any():
    raise ValueError("completed_deliveries must be positive before calculating cost per completion")
if (df["completed_deliveries"] > df["attempted_deliveries"]).any():
    raise ValueError("completed cannot exceed attempted")
if (df["total_cost_pkr"] < 0).any():
    raise ValueError("cost cannot be negative in this fixture")

df["completion_rate"] = df["completed_deliveries"] / df["attempted_deliveries"]
df["cost_per_completed_pkr"] = df["total_cost_pkr"] / df["completed_deliveries"]

# Normalize only within this comparison. Lower cost/support is better.
def benefit(series):
    span = series.max() - series.min()
    return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series - series.min()) / span

def inverse_benefit(series):
    span = series.max() - series.min()
    return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series.max() - series) / span

weights = {"reliability": 0.50, "cost": 0.30, "support": 0.20}
df["score"] = (
    weights["reliability"] * benefit(df["completion_rate"])
    + weights["cost"] * inverse_benefit(df["cost_per_completed_pkr"])
    + weights["support"] * inverse_benefit(df["median_support_hours"])
)

# A missing verified term is a gate, not a hidden scoring penalty.
df["decision_status"] = df["terms_verified"].map(
    {True: "eligible_for_pilot", False: "blocked_pending_terms"}
)

columns = [
    "option", "completion_rate", "cost_per_completed_pkr",
    "median_support_hours", "score", "decision_status"
]
result = df[columns].round(4)
Path("calculated_options.csv").write_text(result.to_csv(index=False), encoding="utf-8")
Path("calculation_record.json").write_text(
    json.dumps({"weights": weights, "rows": result.to_dict(orient="records")}, indent=2),
    encoding="utf-8",
)
print(result.to_string(index=False))

Run python score.py. The code checks impossible values, calculates completion rate and cost per completed delivery, records weights, and blocks an option whose terms are unverified. A high score must not override a failed gate.

The normalization is only illustrative. With two options, small differences can become extreme normalized scores. Report original units beside any score, explain the weights, and never present the score as objective truth.

// concept

Add Documentary and Qualitative Evidence

Suppose the synthetic quotation contains a cancellation clause on page 2, while the interview audio says support response is the team's biggest concern at 00:12–00:18. Extract each as a draft claim with source pointer, then have a human verify it. Keep these categories separate:

  • source fact: what the authorized document/data/audio directly supports;
  • calculation: a reproducible transformation such as completed/attempted;
  • assumption: a chosen weight, threshold, forecast, or proxy;
  • inference: an interpretation joining facts;
  • unknown: evidence not available or not verified;
  • recommendation: the decision owner's proposed action and conditions.

A model may help turn the evidence table into prose, but restrict it to the evidence register. Ask it to list unsupported statements and counterarguments before polishing. Then manually verify every citation and number against the source and calculation output.

// concept

Use a Memo Structure That Exposes Weakness

Write no more than two pages plus appendices:

// template — copy me24 lines
# Decision Memo: [specific decision]

Owner / decision date / review date / memo version

## Recommendation
[action, scope, duration, and conditions]

## Decision rule
[gates, metrics, weights, and threshold]

## Evidence
- [claim] — [E001 rows/filter or E002 page/section]

## Calculations and assumptions
[formula, units, weights, missingness, exclusions]

## Risks, counterevidence, and unknowns
[what could reverse the recommendation]

## Pilot and stop conditions
[what will be measured; who can stop it]

## Source register
[hash, owner, date, permission, pointer]

The recommendation should be conditional when evidence is incomplete: “Run a four-week pilot after terms are verified” is defensible; “Option B is definitely best” is not.

// concept

Sensitivity Analysis: Try to Reverse Your Own Answer

Weights encode priorities, not discovered facts. Recalculate at least three scenarios:

  1. reliability-first: 70% reliability, 20% cost, 10% support;
  2. balanced: 50%, 30%, 20%;
  3. support-first: 40%, 20%, 40%.

Also test one adverse data condition, such as five disputed completions being reclassified, or an unresolved cost being 10% higher. These are hypothetical stress tests, not forecasts. If the recommended eligible option changes easily, report the decision as sensitive and design a pilot to collect the missing evidence. Do not hide instability behind decimal precision.

// failure_cases

Failure Cases That Make a Memo Unreliable

11 cases to diagnose

  • Cherry-picked period

    state why the window was chosen and show material excluded periods.

  • Mismatched populations

    compare similar routes, order types, and service conditions or disclose the limitation.

  • Average hides distribution

    include denominators and appropriate median, range, or segment results.

  • Correlation becomes causation

    a pattern does not prove why it happened.

  • Unverified quotation fact

    distinguish an unsigned or stale fixture from an accepted term.

  • Audio opinion becomes policy

    label the speaker, consent, scope, and whether the statement is fact or preference.

  • False citation

    open every page, row filter, and timestamp; a plausible pointer is not proof.

  • Scoring conceals a gate

    authorization, missing terms, privacy, or safety cannot be traded away for points.

  • AI edits a number

    regenerate tables from code and compare memo values automatically where possible.

  • No dissent

    include the strongest counterargument and evidence that would change the decision.

  • No owner or review date

    recommendations become stale folklore instead of controlled decisions.

// pakistan_angle

Pakistan Angle

Use explicit PKR units and dates in ISO format inside calculations. If a real decision involves USD or another currency, record the exchange-rate source, timestamp, and sensitivity rather than asking AI for a current rate from memory. Segment operational evidence by relevant city, route, language, bandwidth, or payment method only when authorized and statistically meaningful; do not generalize one Lahore or Karachi sample to all Pakistan.

For a small business, the lowest-cost professional approach is often a reversible pilot with clear stop conditions, not a large purchase based on a polished forecast. Minimize customer and rider data: aggregate counts where possible and remove phone numbers, addresses, CNIC information, and private messages from the analytic pack. Contract interpretation, tax, employment, safety, and regulatory questions require current qualified advice; the memo should identify them as review gates rather than manufacture certainty.

// hands_on

Hands-On Exercise: Deliver the Full Decision Pack

8 steps

Create your own synthetic two-option decision using a CSV, a two-page PDF, and a short audio note. Submit:

  1. raw/ with immutable synthetic sources and evidence-manifest.json with hashes and permissions.

  2. DATA-DICTIONARY.md, reproducible cleaning code, quality_report.json, and unresolved-row queue.

  3. claims.csv linking documentary and audio claims to exact pages/sections/timestamps and human review status.

  4. score.py or equivalent with input assertions, formulas, original units, documented weights, and gates.

  5. sensitivity.csv containing at least three weight scenarios and one adverse data scenario.

  6. DECISION-MEMO.md following the template, limited to two pages before appendices.

  7. MEMO-CHECK.md mapping every material number and claim back to a calculation or source, plus the strongest counterargument.

  8. A one-command run that regenerates cleaned data, calculations, and sensitivity results without modifying raw evidence.

// completion_rubric

Completion Rubric

7 grading bands

  • Pass — traceability

    every material claim and number maps to a hashed source pointer or reproducible calculation.

  • Pass — separation

    facts, calculations, assumptions, inference, unknowns, and recommendations are visibly distinct.

  • Pass — data quality

    grain, missingness, duplicates, exclusions, and reconciliation are documented before scoring.

  • Pass — decision discipline

    gates cannot be overridden by scores; owner, deadline, pilot, stop condition, and review date are explicit.

  • Pass — challenge

    sensitivity scenarios and a credible counterargument show what could reverse the choice.

  • Pass — privacy

    all evidence is synthetic and the design minimizes personal fields for future real use.

  • Needs revision

    the memo contains uncited claims, manual calculations that cannot be rerun, invented missing facts, or a permanent recommendation from fragile evidence.

// sources

Sources

// check_yourself

Check yourself

4 questions · answers and options are taken word-for-word from this course

0/4
  1. 1 / 4 · diagnose

    Your work shows this failure mode: “Cherry-picked period.” What does the lesson tell you to do about it?