Lab: Produce an Evidence-Backed Decision Memo

An evidence-backed decision memo states a decision, separates verified facts from calculations and assumptions, shows how each material claim maps to a source, tests whether the recommendation changes under reasonable uncertainty, and names a human owner. AI may organize and challenge the memo; it must not invent missing evidence or make the accountable decision. The deliverable is not “a good answer.” It is a reviewable chain from source to calculation to recommendation.

This capstone combines tabular cleaning and multimodal provenance. Use only synthetic evidence. The scenario below is fictional so no vendor, rate, or market claim should be treated as current.

Frame a Decision That Evidence Can Answer

Weak question: “Which delivery partner is best?”

Testable question: “For our synthetic 90-day Lahore pilot, which option should receive a four-week controlled test, given approved priorities of reliability, cost per completed delivery, and support responsiveness?”

Write the decision owner, deadline, options, constraints, and reversible next step. A pilot recommendation is different from a permanent contract. Define the comparison period and population. If one option handled different routes or order sizes, a raw average may be unfair.

Create an evidence register:

Hash every file and preserve raw and derived versions. Every memo claim must cite an evidence ID plus row/filter, page/section, or timestamp.

Worked Example: Calculate Before Asking for Narrative

Create options.csv:

option,completed_deliveries,attempted_deliveries,total_cost_pkr,median_support_hours,terms_verified
Option A,88,100,30800,5,true
Option B,81,90,27000,2,false

These figures are invented only to demonstrate calculations. Create score.py:

from pathlib import Path
import json
import pandas as pd

df = pd.read_csv("options.csv")
required = {
    "option", "completed_deliveries", "attempted_deliveries",
    "total_cost_pkr", "median_support_hours", "terms_verified"
}
missing = required - set(df.columns)
if missing:
    raise ValueError(f"Missing columns: {sorted(missing)}")

if (df["attempted_deliveries"] <= 0).any():
    raise ValueError("attempted_deliveries must be positive")
if (df["completed_deliveries"] <= 0).any():
    raise ValueError("completed_deliveries must be positive before calculating cost per completion")
if (df["completed_deliveries"] > df["attempted_deliveries"]).any():
    raise ValueError("completed cannot exceed attempted")
if (df["total_cost_pkr"] < 0).any():
    raise ValueError("cost cannot be negative in this fixture")

df["completion_rate"] = df["completed_deliveries"] / df["attempted_deliveries"]
df["cost_per_completed_pkr"] = df["total_cost_pkr"] / df["completed_deliveries"]

# Normalize only within this comparison. Lower cost/support is better.
def benefit(series):
    span = series.max() - series.min()
    return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series - series.min()) / span

def inverse_benefit(series):
    span = series.max() - series.min()
    return pd.Series([1.0] * len(series), index=series.index) if span == 0 else (series.max() - series) / span

weights = {"reliability": 0.50, "cost": 0.30, "support": 0.20}
df["score"] = (
    weights["reliability"] * benefit(df["completion_rate"])
    + weights["cost"] * inverse_benefit(df["cost_per_completed_pkr"])
    + weights["support"] * inverse_benefit(df["median_support_hours"])
)

# A missing verified term is a gate, not a hidden scoring penalty.
df["decision_status"] = df["terms_verified"].map(
    {True: "eligible_for_pilot", False: "blocked_pending_terms"}
)

columns = [
    "option", "completion_rate", "cost_per_completed_pkr",
    "median_support_hours", "score", "decision_status"
]
result = df[columns].round(4)
Path("calculated_options.csv").write_text(result.to_csv(index=False), encoding="utf-8")
Path("calculation_record.json").write_text(
    json.dumps({"weights": weights, "rows": result.to_dict(orient="records")}, indent=2),
    encoding="utf-8",
)
print(result.to_string(index=False))

Run python score.py. The code checks impossible values, calculates completion rate and cost per completed delivery, records weights, and blocks an option whose terms are unverified. A high score must not override a failed gate.

The normalization is only illustrative. With two options, small differences can become extreme normalized scores. Report original units beside any score, explain the weights, and never present the score as objective truth.

Add Documentary and Qualitative Evidence

Suppose the synthetic quotation contains a cancellation clause on page 2, while the interview audio says support response is the team's biggest concern at 00:12–00:18. Extract each as a draft claim with source pointer, then have a human verify it. Keep these categories separate:

• source fact: what the authorized document/data/audio directly supports;
• calculation: a reproducible transformation such as completed/attempted;
• assumption: a chosen weight, threshold, forecast, or proxy;
• inference: an interpretation joining facts;
• unknown: evidence not available or not verified;
• recommendation: the decision owner's proposed action and conditions.

A model may help turn the evidence table into prose, but restrict it to the evidence register. Ask it to list unsupported statements and counterarguments before polishing. Then manually verify every citation and number against the source and calculation output.

Use a Memo Structure That Exposes Weakness

Write no more than two pages plus appendices:

# Decision Memo: [specific decision]

Owner / decision date / review date / memo version

## Recommendation
[action, scope, duration, and conditions]

## Decision rule
[gates, metrics, weights, and threshold]

## Evidence
- [claim] — [E001 rows/filter or E002 page/section]

## Calculations and assumptions
[formula, units, weights, missingness, exclusions]

## Risks, counterevidence, and unknowns
[what could reverse the recommendation]

## Pilot and stop conditions
[what will be measured; who can stop it]

## Source register
[hash, owner, date, permission, pointer]

The recommendation should be conditional when evidence is incomplete: “Run a four-week pilot after terms are verified” is defensible; “Option B is definitely best” is not.

Sensitivity Analysis: Try to Reverse Your Own Answer

Weights encode priorities, not discovered facts. Recalculate at least three scenarios:

1. reliability-first: 70% reliability, 20% cost, 10% support;
2. balanced: 50%, 30%, 20%;
3. support-first: 40%, 20%, 40%.

Also test one adverse data condition, such as five disputed completions being reclassified, or an unresolved cost being 10% higher. These are hypothetical stress tests, not forecasts. If the recommended eligible option changes easily, report the decision as sensitive and design a pilot to collect the missing evidence. Do not hide instability behind decimal precision.

Failure Cases That Make a Memo Unreliable

• Cherry-picked period: state why the window was chosen and show material excluded periods.
• Mismatched populations: compare similar routes, order types, and service conditions or disclose the limitation.
• Average hides distribution: include denominators and appropriate median, range, or segment results.
• Correlation becomes causation: a pattern does not prove why it happened.
• Unverified quotation fact: distinguish an unsigned or stale fixture from an accepted term.
• Audio opinion becomes policy: label the speaker, consent, scope, and whether the statement is fact or preference.
• False citation: open every page, row filter, and timestamp; a plausible pointer is not proof.
• Scoring conceals a gate: authorization, missing terms, privacy, or safety cannot be traded away for points.
• AI edits a number: regenerate tables from code and compare memo values automatically where possible.
• No dissent: include the strongest counterargument and evidence that would change the decision.
• No owner or review date: recommendations become stale folklore instead of controlled decisions.

🇵🇰 Pakistan Angle

Use explicit PKR units and dates in ISO format inside calculations. If a real decision involves USD or another currency, record the exchange-rate source, timestamp, and sensitivity rather than asking AI for a current rate from memory. Segment operational evidence by relevant city, route, language, bandwidth, or payment method only when authorized and statistically meaningful; do not generalize one Lahore or Karachi sample to all Pakistan.

For a small business, the lowest-cost professional approach is often a reversible pilot with clear stop conditions, not a large purchase based on a polished forecast. Minimize customer and rider data: aggregate counts where possible and remove phone numbers, addresses, CNIC information, and private messages from the analytic pack. Contract interpretation, tax, employment, safety, and regulatory questions require current qualified advice; the memo should identify them as review gates rather than manufacture certainty.

Hands-On Exercise: Deliver the Full Decision Pack

Create your own synthetic two-option decision using a CSV, a two-page PDF, and a short audio note. Submit:

1. raw/ with immutable synthetic sources and evidence-manifest.json with hashes and permissions.
2. DATA-DICTIONARY.md, reproducible cleaning code, quality_report.json, and unresolved-row queue.
3. claims.csv linking documentary and audio claims to exact pages/sections/timestamps and human review status.
4. score.py or equivalent with input assertions, formulas, original units, documented weights, and gates.
5. sensitivity.csv containing at least three weight scenarios and one adverse data scenario.
6. DECISION-MEMO.md following the template, limited to two pages before appendices.
7. MEMO-CHECK.md mapping every material number and claim back to a calculation or source, plus the strongest counterargument.
8. A one-command run that regenerates cleaned data, calculations, and sensitivity results without modifying raw evidence.

Completion Rubric

• Pass — traceability: every material claim and number maps to a hashed source pointer or reproducible calculation.
• Pass — separation: facts, calculations, assumptions, inference, unknowns, and recommendations are visibly distinct.
• Pass — data quality: grain, missingness, duplicates, exclusions, and reconciliation are documented before scoring.
• Pass — decision discipline: gates cannot be overridden by scores; owner, deadline, pilot, stop condition, and review date are explicit.
• Pass — challenge: sensitivity scenarios and a credible counterargument show what could reverse the choice.
• Pass — privacy: all evidence is synthetic and the design minimizes personal fields for future real use.
• Needs revision: the memo contains uncited claims, manual calculations that cannot be rerun, invented missing facts, or a permanent recommendation from fragile evidence.

Sources

• pandas: User guide
• OpenAI: Structured model outputs
• OpenAI: File inputs
• NIST: AI Risk Management Framework
• NIST: AI RMF Playbook

Key takeaway: A decision memo is defensible when every claim traces to authorized evidence, every number can be regenerated, assumptions and unknowns remain visible, sensitivity tests challenge the recommendation, and a named human owns the reversible next step.