Retries are safe only when the failure class and side-effect state are known. Repeating a model turn, payment request, email send, or file write can duplicate harm. A swarm needs per-task idempotency, bounded retry, circuit breaking, dead-letter review, and run-level cancellation.
After this lesson, you can design a failure matrix and recovery drill.
Classify Before Retrying
| Class | Example | Response |
|---|---|---|
| transient | timeout, rate limit, temporary outage | bounded backoff with same idempotency key |
| permanent input | invalid schema, missing required record | stop and correct input |
| permission | forbidden scope | stop; never bypass |
| policy | requested action prohibited | stop and record |
| ambiguous effect | timeout after send | query authoritative state before retry |
| quality | output fails evaluator | limited repair or human review |
| budget | run cap reached | cancel remaining work |
Do not let a model classify its own authorization failure as transient.
Use Backoff and Jitter
Retry temporary failures after increasing delays with randomness, respecting provider guidance such as Retry-After. Cap attempts and elapsed time. A circuit breaker pauses new calls when a dependency is failing broadly; queued work retains its identity for later recovery.
Preserve Run State
Store task attempt, tool event IDs, last committed checkpoint, error code, next eligible time, and owner. On cancellation, stop issuing new tasks and signal workers. A dead-letter queue is not a graveyard: it needs a dashboard, access controls, replay procedure, and age target.
Worked Example
Ten workers generate approved product-copy drafts. The catalog API begins returning 429. Workers record the same task IDs, respect retry timing, and the circuit breaker pauses catalog calls. No worker invents product details from memory.
One email tool times out after submission. The workflow queries the email provider using its idempotency/reference ID; it finds the message was accepted and marks the effect complete. It does not send again. Three permanent schema failures enter review with redacted evidence. The run stops when its cost budget is reached, leaving resumable tasks rather than spawning replacements.
Failure Cases to Diagnose
- Retry wraps the whole workflow: resume from checkpoints.
- New idempotency key per attempt: retain logical action identity.
- Timeout assumed failure: reconcile with authoritative state.
- Every worker hammers a down service: use shared rate and circuit controls.
- Dead letters never reviewed: assign owner and age target.
- Cancellation does not reach workers: propagate and enforce it.
- Error log contains prompts and secrets: redact and restrict.
🇵🇰 Pakistan Angle
Local bank, wallet, courier, SMS, and government services may have maintenance or inconsistent latency. Treat each integration according to its official contract. If the state is uncertain, pause and reconcile; never issue another payment, filing, or customer promise merely because a request timed out.
Communicate outages honestly in PKT with case references and realistic next updates. Avoid infinite retry that consumes foreign-currency model or API spend during a regional network problem. Budget and circuit controls protect both the client and customer.
Hands-On Exercise
- Build the failure-class table for every tool.
- Define retry delays, caps, idempotency, and reconciliation.
- Add circuit breaker, cancellation, and dead-letter ownership.
- Simulate rate limit, timeout-after-effect, invalid input, and budget exhaustion.
- Replay one recovered task and prove no duplicate effect.
Done means: every failure stops, retries, reconciles, or escalates according to a predeclared rule, and replay cannot duplicate a committed action.
Completion Rubric
- Failure classes map to different responses.
- Transient retries are bounded and jittered.
- Idempotency survives attempts and worker changes.
- Ambiguous effects reconcile before retry.
- Circuit, cancellation, and dead-letter paths are operated.
- Logs are useful without exposing secrets.
Sources
- AWS — Builders’ Library: Timeouts, retries and backoff with jitter
- Google Cloud — Retry strategy
- OWASP — Logging Cheat Sheet
Key takeaway: retries are a state-and-effect protocol, not a loop around an error; classify, reconcile, deduplicate, cap, and escalate.