An agent framework packages model calls, tools, state, routing, tracing, and sometimes deployment. A managed platform may reduce setup; a library may offer control; plain application code may be clearest for a small workflow. Compare evidence, not feature-count marketing.
After this lesson, you can run a framework evaluation using one fixed task and a weighted decision record.
Start With Requirements
List non-negotiables before products:
| Area | Questions |
|---|---|
| model/runtime | which providers and deployment regions are required? |
| tools | schemas, auth, approval, timeout, idempotency? |
| state | durable checkpoints, versioning, deletion, tenancy? |
| observability | traces, redaction, export, retention? |
| reliability | retry, resume, queue, concurrency, rollback? |
| security | SSO, roles, secrets, audit, data controls? |
| operations | hosting, upgrades, on-call, support, exit path? |
| economics | model, platform, storage, egress, and staff cost? |
Use current official documentation for candidates; capabilities and pricing change.
Prototype One Representative Task
Give every option the same task, tools, data, and evaluation set. Measure completion accuracy, invalid actions blocked, tool calls, latency, model tokens/cost where available, recovery after interruption, trace usefulness, and engineering time.
Do not compare a vendor demo with a production implementation. Include failure tests: tool timeout, malformed model output, duplicate event, missing permission, state resume, and sensitive-data redaction.
Decide Build, Library, or Managed
- Plain code: best when flow is small, deterministic, and needs precise control.
- Framework/library: useful when tested graph/state/tool abstractions remove repeated work.
- Managed platform: useful when hosted tracing, evaluation, security, and operations justify dependency and cost.
Document lock-in: stored state format, proprietary tools, trace export, model portability, and migration effort.
Worked Example
A Karachi software agency evaluates an invoice-intake assistant. It must extract allowed fields, match a purchase order, prepare an exception case, and never approve payment. The team prototypes plain TypeScript, one open-source graph library, and one managed service.
Weights are correctness 30%, safety 25%, operability 20%, cost 15%, and developer experience 10%. The plain implementation wins because the flow has four deterministic states and the managed platform adds little. The decision is revisited if workflows or teams grow. No framework is declared universally superior.
Failure Cases to Diagnose
- Choosing from a social-media benchmark: reproduce the actual task.
- Ignoring failure recovery: test restart and duplicate delivery.
- Token price treated as total cost: include engineering and operations.
- Framework handles auth assumed: inspect where authorization is enforced.
- No data export path: test state and trace portability.
- One huge prototype: compare the smallest representative slice.
- Version pinned nowhere: record versions and upgrade policy.
🇵🇰 Pakistan Angle
Include payment method availability, foreign-currency billing, tax/accounting treatment, support timezone, and data-transfer requirements in the decision. Do not recommend a tool only because a free tier exists today; calculate the expected PKR monthly cost under low, expected, and stress usage.
Connectivity and local hiring matter. A simpler system the team can test and recover may beat a sophisticated framework dependent on one specialist. For client delivery, disclose third-party processors and obtain authority before sending real business data.
Hands-On Exercise
- Write the requirement matrix and weights.
- Select one representative task and fixed evaluation set.
- Compare plain code, a library, and managed option.
- Run six failure tests.
- Publish a decision record with revisit conditions.
Done means: the choice is reproducible from requirements, measured behavior, full cost, and exit risk.
Completion Rubric
- Requirements precede product selection.
- Candidates receive the same task and evaluation.
- Safety and recovery tests affect the score.
- Total cost and lock-in are visible.
- Versions and official sources are recorded.
- Revisit conditions prevent permanent folklore.
Sources
- OpenAI — Agents guide
- Anthropic — Building effective agents
- NIST — Secure Software Development Framework
Key takeaway: choose the smallest runtime that meets measured correctness, safety, state, and operating needs; framework sophistication is not an outcome.