Module 03 · Swarm Orchestration
Handling Failures and Retries Across an Agent Swarm
Open lesson + course map
On this lesson
Course outline
Module 1 · Agent Fundamentals
Module 2 · Agent Frameworks
Module 3 · Swarm Orchestration
Module 4 · Operational Guardrails
Module 5 · Shipping a Real Agent System
Retries are safe only when the failure class and side-effect state are known. Repeating a model turn, payment request, email send, or file write can duplicate harm. A swarm needs per-task idempotency, bounded retry, circuit breaking, dead-letter review, and run-level cancellation.
After this lesson, you can design a failure matrix and recovery drill.
// concept
Classify Before Retrying
| Class | Example | Response |
|---|---|---|
| transient | timeout, rate limit, temporary outage | bounded backoff with same idempotency key |
| permanent input | invalid schema, missing required record | stop and correct input |
| permission | forbidden scope | stop; never bypass |
| policy | requested action prohibited | stop and record |
| ambiguous effect | timeout after send | query authoritative state before retry |
| quality | output fails evaluator | limited repair or human review |
| budget | run cap reached | cancel remaining work |
Do not let a model classify its own authorization failure as transient.
// concept
Use Backoff and Jitter
Retry temporary failures after increasing delays with randomness, respecting provider guidance such as Retry-After. Cap attempts and elapsed time. A circuit breaker pauses new calls when a dependency is failing broadly; queued work retains its identity for later recovery.
// concept
Preserve Run State
Store task attempt, tool event IDs, last committed checkpoint, error code, next eligible time, and owner. On cancellation, stop issuing new tasks and signal workers. A dead-letter queue is not a graveyard: it needs a dashboard, access controls, replay procedure, and age target.
// worked_example
Worked Example
Ten workers generate approved product-copy drafts. The catalog API begins returning 429. Workers record the same task IDs, respect retry timing, and the circuit breaker pauses catalog calls. No worker invents product details from memory.
One email tool times out after submission. The workflow queries the email provider using its idempotency/reference ID; it finds the message was accepted and marks the effect complete. It does not send again. Three permanent schema failures enter review with redacted evidence. The run stops when its cost budget is reached, leaving resumable tasks rather than spawning replacements.
// failure_cases
Failure Cases to Diagnose
7 cases to diagnose
Retry wraps the whole workflow
resume from checkpoints.
New idempotency key per attempt
retain logical action identity.
Timeout assumed failure
reconcile with authoritative state.
Every worker hammers a down service
use shared rate and circuit controls.
Dead letters never reviewed
assign owner and age target.
Cancellation does not reach workers
propagate and enforce it.
Error log contains prompts and secrets
redact and restrict.
// pakistan_angle
Pakistan Angle
Local bank, wallet, courier, SMS, and government services may have maintenance or inconsistent latency. Treat each integration according to its official contract. If the state is uncertain, pause and reconcile; never issue another payment, filing, or customer promise merely because a request timed out.
Communicate outages honestly in PKT with case references and realistic next updates. Avoid infinite retry that consumes foreign-currency model or API spend during a regional network problem. Budget and circuit controls protect both the client and customer.
// hands_on
Hands-On Exercise
5 steps
Build the failure-class table for every tool.
Define retry delays, caps, idempotency, and reconciliation.
Add circuit breaker, cancellation, and dead-letter ownership.
Simulate rate limit, timeout-after-effect, invalid input, and budget exhaustion.
Replay one recovered task and prove no duplicate effect.
// completion_rubric
Completion Rubric
6 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself
// check_yourself
Check yourself
4 questions · answers and options are taken word-for-word from this course
1 / 4 · diagnose
Your work shows this failure mode: “New idempotency key per attempt.” What does the lesson tell you to do about it?