Direct answer
Reliable AI workflows remember what has happened, retry only operations that are safe to repeat, prevent duplicate side effects, and pause for human approval before consequential actions. State records the workflow's durable position. Retries recover from temporary failures. Idempotency makes repeated processing produce one intended effect. Human approval transfers a clearly described decision to an accountable person before the system acts.
These are application responsibilities. A stronger prompt does not fix a duplicate payment, a lost job after a server restart, or an approval button that authorizes different data from what the reviewer saw.
Model the workflow as states
Do not represent a job with only done: true/false. Use named states and allowed transitions. A purchase-order assistant might use:
RECEIVED -> VALIDATED -> RESEARCHED -> DRAFTED -> AWAITING_APPROVAL
AWAITING_APPROVAL -> APPROVED -> SENDING -> SENT
AWAITING_APPROVAL -> REJECTED
any non-final state -> FAILED_RETRYABLE or FAILED_FINAL
Store the job ID, current state, input reference, authenticated requester, workflow version, source versions, tool results, approval record, retry count, timestamps, and idempotency keys. Store only the data needed for the purpose, and define retention. State must survive a process crash; an in-memory variable is useful for a lesson but insufficient for production.
Validate every transition. RECEIVED -> SENT should be impossible. An approval made against draft version 2 should not authorize silently changed version 3. If a reviewer changes a material field, produce a new draft hash and request approval again.
Retries are for temporary failures
Retry network timeouts, rate limits, and transient service errors when the operation is documented as retry-safe. Do not retry invalid input, forbidden access, an exhausted budget, or a policy rejection. Use exponential backoff with jitter so many workers do not retry simultaneously:
delay = min(max_delay, base_delay * 2^attempt) + random_jitter
Cap attempts and elapsed time. After the cap, move the job to a visible dead-letter or manual-review queue with enough context to diagnose it. An endless retry loop is neither resilient nor cheap.
A timeout is ambiguous: the remote service may have completed the action while your client missed the response. That is why retries and idempotency must be designed together.
Idempotency prevents duplicate effects
An idempotency key identifies one business action, not one HTTP attempt. For example, send-purchase-order:PO-2026-014:v2 represents sending version 2 of one approved purchase order. Store the key and result. If the same webhook arrives twice or a worker restarts, return the recorded result instead of sending again.
Generate the key from stable business identifiers or create it when the job begins. Do not use the current timestamp for every retry; that turns each retry into a new action. For database work, use a unique constraint or transactional outbox pattern rather than hoping two workers do not race.
Worked build: a supplier purchase-order assistant
A Lahore electronics retailer uploads a restock sheet. The workflow validates rows, retrieves approved supplier terms, drafts a purchase order, and asks a purchasing manager to approve. It may never email the supplier before approval.
This minimal JavaScript example demonstrates the control logic with in-memory storage. Replace the Map with durable transactional storage in a real service:
import crypto from "node:crypto";
const jobs = new Map();
const effects = new Map();
function hash(value) {
return crypto.createHash("sha256").update(JSON.stringify(value)).digest("hex");
}
function cloneAndFreeze(value) {
const clone = structuredClone(value);
const freeze = (item) => {
if (item && typeof item === "object" && !Object.isFrozen(item)) {
Object.values(item).forEach(freeze);
Object.freeze(item);
}
return item;
};
return freeze(clone);
}
function createJob(id, draft) {
const immutableDraft = cloneAndFreeze(draft);
const job = {
id,
state: "AWAITING_APPROVAL",
draft: immutableDraft,
draftHash: hash(immutableDraft),
attempts: 0,
};
jobs.set(id, job);
return structuredClone(job);
}
// authContext must come from trusted server-side authentication middleware,
// never from reviewer fields submitted in request JSON.
function approve(jobId, authContext, reviewedHash) {
const job = jobs.get(jobId);
if (!job || job.state !== "AWAITING_APPROVAL") throw new Error("Invalid state");
if (!authContext?.userId || !authContext.roles?.includes("purchase-approver")) {
throw new Error("Authenticated purchase approver required");
}
const currentHash = hash(job.draft);
if (currentHash !== job.draftHash || reviewedHash !== currentHash) {
throw new Error("Draft changed; review again");
}
job.state = "APPROVED";
job.approval = { reviewerId: authContext.userId, reviewedHash: currentHash, at: new Date().toISOString() };
}
async function sendOnce(jobId, sendEmail) {
const job = jobs.get(jobId);
if (!job || job.state !== "APPROVED") throw new Error("Approval required");
const currentHash = hash(job.draft);
if (currentHash !== job.draftHash || currentHash !== job.approval.reviewedHash) {
throw new Error("Approved payload changed; execution blocked");
}
const key = `send-po:${job.id}:${currentHash}`;
if (effects.has(key)) return effects.get(key);
job.state = "SENDING";
const result = await sendEmail(structuredClone(job.draft), { idempotencyKey: key });
effects.set(key, result);
job.state = "SENT";
return result;
}
This illustrates the invariants, but it still has a production gap: the process could crash after sendEmail succeeds and before effects.set. A remote email service that accepts the same idempotency key can close that gap. Otherwise, use a durable outbox and reconcile ambiguous sends rather than blindly trying again.
Design an approval people can actually review
An approval screen must show the proposed action, recipient, amount or scope, evidence, assumptions, policy flags, and what will happen after approval. “Approve agent output?” is not informed review. The reviewer should see a readable draft and a machine-stable hash or version behind it.
Record:
- authenticated reviewer identity and role;
- exact draft version reviewed;
- approval or rejection time;
- any comment or required change;
- policy basis for requiring approval;
- the final execution result.
Approvals should expire when the underlying facts can change. A stock order approved on Monday should not remain executable after quantities or prices change on Friday. Define expiry and revalidation rules.
Use approval before irreversible or high-impact actions: sending messages to external people, publishing content, changing permissions, deleting data, committing code, placing orders, issuing refunds, or submitting official forms. For low-risk drafts, post-action review may be enough. Match the control to consequence.
OpenAI's agent guidance supports resumable approval flows and guardrails, but an SDK does not decide your business policy. You still define which tool calls pause, who may approve, and how the approved payload is bound to execution.
Failure cases to force
- A webhook arrives three times and creates three jobs.
- A rate-limit response triggers immediate retries from hundreds of workers.
- A send call times out after success and creates a duplicate on retry.
- A reviewer approves a summary, but the hidden recipient or amount differs.
- The draft changes after approval but retains the approved state.
- A rejected job is accidentally picked up by a generic retry worker.
- Two workers execute the same approved action concurrently.
- An agent writes “approved” in its own output and application code mistakes that for human authorization.
- State lives only in conversation history and disappears after compaction or restart.
- Logs contain complete supplier banking or personal contact information without need.
🇵🇰 Pakistan Angle
Intermittent internet and load-shedding make duplicate-safe workflows practical necessities in Pakistan. A user may tap “submit” twice when the first response is slow. A worker may restart midway through a JazzCash-related reconciliation, COD order update, or WhatsApp message. Design the system so reconnecting and retrying do not repeat the business action.
Approval interfaces should work on a phone, show PKR amounts clearly, use Pakistan Standard Time, and identify the organization and recipient without exposing unnecessary personal data. Do not rely on a forwarded WhatsApp “okay” unless your organization has deliberately defined and authenticated that approval channel. For payments, contracts, employment decisions, or regulated activity, use the authorized process and professional advice; an AI workflow should prepare evidence, not impersonate authority.
Hands-on deliverable
Build a reliability pack for one workflow with an external side effect:
- a state diagram with allowed and forbidden transitions;
- a durable-state schema including workflow and source versions;
- a retry table listing transient, permanent, and ambiguous errors;
- an idempotency-key design and duplicate-webhook test;
- an approval screen mock showing action, evidence, reviewer, version, and expiry;
- tests for crash-before-call, crash-after-call, double approval, changed draft, and concurrent workers;
- a dead-letter/manual-review procedure with an owner and response time.
Completion rubric
- State has named transitions and survives process restarts in the production design.
- Retries are limited to classified transient or explicitly safe operations.
- Idempotency keys represent one business action across all attempts.
- Duplicate delivery and concurrent execution cannot create duplicate effects.
- Approval is tied to authenticated reviewer, exact payload version, and expiry.
- The agent cannot authorize its own consequential action.
- Ambiguous outcomes go to reconciliation rather than blind retry.
- Logs and state minimize sensitive data.
Key takeaway: Reliability comes from durable state and enforceable invariants: retry safely, execute each intended effect once, and bind human approval to the exact action that will occur.