Module 03 · Data and Multimodal Work
Lab: Produce an Evidence-Backed Decision Memo
Open lesson + course map
On this lesson
Course outline
Module 1 · Technical Foundations
Module 2 · Structured Outputs and Tools
Module 3 · Data and Multimodal Work
Module 4 · Grounded Knowledge Systems
Module 5 · Agents and Reliability
Module 6 · Ship the Professional Capstone
An evidence-backed decision memo states a decision, separates verified facts from calculations and assumptions, shows how each material claim maps to a source, tests whether the recommendation changes under reasonable uncertainty, and names a human owner. AI may organize and challenge the memo; it must not invent missing evidence or make the accountable decision. The deliverable is not “a good answer.” It is a reviewable chain from source to calculation to recommendation.
This capstone combines tabular cleaning and multimodal provenance. Use only synthetic evidence. The scenario below is fictional so no vendor, rate, or market claim should be treated as current.
// concept
Frame a Decision That Evidence Can Answer
Weak question: “Which delivery partner is best?”
Testable question: “For our synthetic 90-day Lahore pilot, which option should receive a four-week controlled test, given approved priorities of reliability, cost per completed delivery, and support responsiveness?”
Write the decision owner, deadline, options, constraints, and reversible next step. A pilot recommendation is different from a permanent contract. Define the comparison period and population. If one option handled different routes or order sizes, a raw average may be unfair.
Create an evidence register:
| Evidence ID | Source | Scope/date | Owner | Material use | Limitation |
|---|---|---|---|---|---|
| E001 | synthetic deliveries.csv | 90-day lab fixture | learner | completion and cost metrics | not real operations |
| E002 | synthetic quotation PDF | lab version 1 | learner | terms and exclusions | unsigned fixture |
| E003 | synthetic interview audio | 35 seconds | learner | support concerns | one fictional speaker |
Hash every file and preserve raw and derived versions. Every memo claim must cite an evidence ID plus row/filter, page/section, or timestamp.
// worked_example
Worked Example: Calculate Before Asking for Narrative
Create options.csv:
option,completed_deliveries,attempted_deliveries,total_cost_pkr,median_support_hours,terms_verified
Option A,88,100,30800,5,true
Option B,81,90,27000,2,falseThese figures are invented only to demonstrate calculations. Create score.py:
from pathlib import Path
import json
import pandas as pd
df = pd.read_csv("options.csv")
required = {
"option", "completed_deliveries", "attempted_deliveries",
"total_cost_pkr", "median_support_hours", "terms_verified"
}
missing = required - set(df.columns)
if missing:
raise ValueError(f"Missing columns: {sorted(missing)}")
if (df["attempted_deliveries"] <= 0).any():
raise ValueError("attempted_deliveries must be positive")
if (df["completed_deliveries"] <= 0).any():
raise ValueError("completed_deliveries must be positive before calculating cost per completion")
if (df["completed_deliveries"] > df["attempted_deliveries"]).any():
raise ValueError("completed cannot exceed attempted")
if (df["total_cost_pkr"] < 0).any():
raise ValueError("cost cannot be negative in this fixture")
df["completion_rate"] = df["completed_deliveries"] / df["attempted_deliveries"]
df["cost_per_completed_pkr"] = df["total_cost_pkr"] / df["completed_deliveries"]
# Normalize only within this comparison. Lower cost/support is better.
def benefit(series):
span = series.max() - series.min()
return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series - series.min()) / span
def inverse_benefit(series):
span = series.max() - series.min()
return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series.max() - series) / span
weights = {"reliability": 0.50, "cost": 0.30, "support": 0.20}
df["score"] = (
weights["reliability"] * benefit(df["completion_rate"])
+ weights["cost"] * inverse_benefit(df["cost_per_completed_pkr"])
+ weights["support"] * inverse_benefit(df["median_support_hours"])
)
# A missing verified term is a gate, not a hidden scoring penalty.
df["decision_status"] = df["terms_verified"].map(
{True: "eligible_for_pilot", False: "blocked_pending_terms"}
)
columns = [
"option", "completion_rate", "cost_per_completed_pkr",
"median_support_hours", "score", "decision_status"
]
result = df[columns].round(4)
Path("calculated_options.csv").write_text(result.to_csv(index=False), encoding="utf-8")
Path("calculation_record.json").write_text(
json.dumps({"weights": weights, "rows": result.to_dict(orient="records")}, indent=2),
encoding="utf-8",
)
print(result.to_string(index=False))Run python score.py. The code checks impossible values, calculates completion rate and cost per completed delivery, records weights, and blocks an option whose terms are unverified. A high score must not override a failed gate.
The normalization is only illustrative. With two options, small differences can become extreme normalized scores. Report original units beside any score, explain the weights, and never present the score as objective truth.
// concept
Add Documentary and Qualitative Evidence
Suppose the synthetic quotation contains a cancellation clause on page 2, while the interview audio says support response is the team's biggest concern at 00:12–00:18. Extract each as a draft claim with source pointer, then have a human verify it. Keep these categories separate:
- source fact: what the authorized document/data/audio directly supports;
- calculation: a reproducible transformation such as completed/attempted;
- assumption: a chosen weight, threshold, forecast, or proxy;
- inference: an interpretation joining facts;
- unknown: evidence not available or not verified;
- recommendation: the decision owner's proposed action and conditions.
A model may help turn the evidence table into prose, but restrict it to the evidence register. Ask it to list unsupported statements and counterarguments before polishing. Then manually verify every citation and number against the source and calculation output.
// concept
Use a Memo Structure That Exposes Weakness
Write no more than two pages plus appendices:
# Decision Memo: [specific decision]
Owner / decision date / review date / memo version
## Recommendation
[action, scope, duration, and conditions]
## Decision rule
[gates, metrics, weights, and threshold]
## Evidence
- [claim] — [E001 rows/filter or E002 page/section]
## Calculations and assumptions
[formula, units, weights, missingness, exclusions]
## Risks, counterevidence, and unknowns
[what could reverse the recommendation]
## Pilot and stop conditions
[what will be measured; who can stop it]
## Source register
[hash, owner, date, permission, pointer]The recommendation should be conditional when evidence is incomplete: “Run a four-week pilot after terms are verified” is defensible; “Option B is definitely best” is not.
// concept
Sensitivity Analysis: Try to Reverse Your Own Answer
Weights encode priorities, not discovered facts. Recalculate at least three scenarios:
- reliability-first: 70% reliability, 20% cost, 10% support;
- balanced: 50%, 30%, 20%;
- support-first: 40%, 20%, 40%.
Also test one adverse data condition, such as five disputed completions being reclassified, or an unresolved cost being 10% higher. These are hypothetical stress tests, not forecasts. If the recommended eligible option changes easily, report the decision as sensitive and design a pilot to collect the missing evidence. Do not hide instability behind decimal precision.
// failure_cases
Failure Cases That Make a Memo Unreliable
11 cases to diagnose
Cherry-picked period
state why the window was chosen and show material excluded periods.
Mismatched populations
compare similar routes, order types, and service conditions or disclose the limitation.
Average hides distribution
include denominators and appropriate median, range, or segment results.
Correlation becomes causation
a pattern does not prove why it happened.
Unverified quotation fact
distinguish an unsigned or stale fixture from an accepted term.
Audio opinion becomes policy
label the speaker, consent, scope, and whether the statement is fact or preference.
False citation
open every page, row filter, and timestamp; a plausible pointer is not proof.
Scoring conceals a gate
authorization, missing terms, privacy, or safety cannot be traded away for points.
AI edits a number
regenerate tables from code and compare memo values automatically where possible.
No dissent
include the strongest counterargument and evidence that would change the decision.
No owner or review date
recommendations become stale folklore instead of controlled decisions.
// pakistan_angle
Pakistan Angle
Use explicit PKR units and dates in ISO format inside calculations. If a real decision involves USD or another currency, record the exchange-rate source, timestamp, and sensitivity rather than asking AI for a current rate from memory. Segment operational evidence by relevant city, route, language, bandwidth, or payment method only when authorized and statistically meaningful; do not generalize one Lahore or Karachi sample to all Pakistan.
For a small business, the lowest-cost professional approach is often a reversible pilot with clear stop conditions, not a large purchase based on a polished forecast. Minimize customer and rider data: aggregate counts where possible and remove phone numbers, addresses, CNIC information, and private messages from the analytic pack. Contract interpretation, tax, employment, safety, and regulatory questions require current qualified advice; the memo should identify them as review gates rather than manufacture certainty.
// hands_on
Hands-On Exercise: Deliver the Full Decision Pack
8 steps
Create your own synthetic two-option decision using a CSV, a two-page PDF, and a short audio note. Submit:
raw/with immutable synthetic sources andevidence-manifest.jsonwith hashes and permissions.DATA-DICTIONARY.md, reproducible cleaning code,quality_report.json, and unresolved-row queue.claims.csvlinking documentary and audio claims to exact pages/sections/timestamps and human review status.score.pyor equivalent with input assertions, formulas, original units, documented weights, and gates.sensitivity.csvcontaining at least three weight scenarios and one adverse data scenario.DECISION-MEMO.mdfollowing the template, limited to two pages before appendices.MEMO-CHECK.mdmapping every material number and claim back to a calculation or source, plus the strongest counterargument.A one-command run that regenerates cleaned data, calculations, and sensitivity results without modifying raw evidence.
// completion_rubric
Completion Rubric
7 grading bands
- Pass — traceability
every material claim and number maps to a hashed source pointer or reproducible calculation.
- Pass — separation
facts, calculations, assumptions, inference, unknowns, and recommendations are visibly distinct.
- Pass — data quality
grain, missingness, duplicates, exclusions, and reconciliation are documented before scoring.
- Pass — decision discipline
gates cannot be overridden by scores; owner, deadline, pilot, stop condition, and review date are explicit.
- Pass — challenge
sensitivity scenarios and a credible counterargument show what could reverse the choice.
- Pass — privacy
all evidence is synthetic and the design minimizes personal fields for future real use.
- Needs revision
the memo contains uncited claims, manual calculations that cannot be rerun, invented missing facts, or a permanent recommendation from fragile evidence.
// sources
Sources
// check_yourself
Check yourself
4 questions · answers and options are taken word-for-word from this course
1 / 4 · diagnose
Your work shows this failure mode: “Cherry-picked period.” What does the lesson tell you to do about it?