Production tools must tell the host whether a failure is retryable, permanent, unauthorized, conflicted, or uncertain after a possible effect. Retrying everything creates duplicate actions and outage storms.
Define a Safe Error Model
{
"ok": false,
"error": {
"code": "RATE_LIMITED",
"retryable": true,
"retry_after_ms": 2000,
"correlation_id": "req_72"
}
}
Do not return stack traces, tokens, SQL, internal paths, or private upstream payloads. Log safe detail under restricted access and give the client a correlation ID.
Classify validation, not found, conflict, unauthenticated, forbidden, rate-limited, temporary unavailable, timeout-unknown, and internal failure.
Control Load
Apply per-identity and per-tenant rate limits, concurrency caps, queue limits, timeouts, and request-size limits. Respect upstream Retry-After guidance. Use bounded exponential backoff with jitter for transient reads.
For writes, require idempotency. If a timeout occurs after submission, query authoritative state before retry. Use circuit breaking during broad upstream failure and dead-letter review for exhausted work.
Worked Example
A courier quote tool allows five concurrent requests per tenant and validates city/weight before the upstream call. When upstream returns 429, the server returns a controlled retry delay. The host retries twice with the same request identity.
A booking creation times out. The server queries by idempotency key, finds the booking, and returns it instead of creating another. Metrics show rate-limit count, queue age, p90 latency, and error class without customer address.
Failure Cases to Diagnose
- Model decides retry timing: enforce in host/tool code.
- New idempotency key per attempt: preserve logical identity.
- Global limit lets one tenant monopolize: add fair per-tenant quotas.
- Queue grows without bound: cap and shed safely.
- 403 retried: authorization failures require correction.
- Logs leak upstream response: redact and minimize.
Operator Note
Publish a small service-level contract for tool consumers: maximum input size, typical timeout, concurrency behavior, error codes, retry obligations, and idempotency support. Version that contract and exercise it in load tests. A client should never guess whether it may retry a write. During overload, reject early with a controlled response instead of accepting unlimited work that will expire unseen.
🇵🇰 Pakistan Angle
Expect intermittent courier, wallet, bank, and government endpoints. Do not replace an unavailable official service with scraping or invented status. Provide a case reference and honest pending state.
Foreign API cost is another quota. Limit tenant spend and convert estimates to PKR using a dated assumption; never let retries silently consume the client’s budget.
Hands-On Exercise
- Define the error taxonomy.
- Add identity/tenant rates, concurrency, queue, and timeout.
- Implement stable write idempotency.
- Test 429, 403, timeout-after-effect, overload, and circuit open.
- Verify redacted metrics and logs.
Completion Rubric
- Errors are typed and safe.
- Retry policy is code-enforced and bounded.
- Writes reconcile and deduplicate.
- Limits protect tenants and dependencies.
- Queue/circuit/dead-letter paths operate.
- Logs reveal no sensitive payload.
Sources
- Google Cloud — Retry strategy
- AWS — Timeouts, retries and backoff with jitter
- OWASP — Logging Cheat Sheet
Key takeaway: encode failure class, fairness, backoff, idempotency, and reconciliation in software so the model cannot turn an outage into duplicate effects.