Observability answers: what run happened, which version acted, what tools were requested, what effects committed, why it stopped, and how to recover. Logging every prompt and private record is not observability; it is often a data leak.
After this lesson, you can define structured events, trace a run, and redact sensitive content.
Use a Correlation Model
Generate run_id, task_id, trace_id, and stable effect/idempotency IDs. Emit structured events:
{
"event": "tool.completed",
"run_id": "run_72",
"task_id": "task_19",
"tool": "create_quote_draft",
"tool_version": "3",
"result": "success",
"duration_ms": 418,
"attempt": 1,
"effect_id": "quote:D-44",
"recorded_at": "2026-07-19T10:00:00Z"
}
Record model/provider identifier, prompt/policy version, usage, evaluator version, approval reference, state transition, error class, and stop reason where relevant.
Redact at Collection
Do not log secrets, tokens, authorization headers, OTPs, passwords, full payment data, or unnecessary identity fields. Prefer references and hashes over raw phone numbers, emails, documents, and customer messages. Restrict access and retention, protect log integrity, and record administrative access.
Keep a separately controlled debug mode for approved synthetic environments. “We may need it later” is not a retention purpose.
Define Service Signals
Track:
- task success and evaluator pass rate;
- invalid or denied tool calls;
- model/tool latency percentiles;
- retry, timeout, dead-letter, and circuit state;
- queue depth and oldest task age;
- tokens/cost per completed outcome;
- human approval wait and rejection reasons;
- duplicate effect prevented;
- drift by workflow/model/version.
Alert on actionability, not noise. Every alert needs owner, threshold rationale, runbook, and test.
Worked Example
A support-draft agent shows a normal completion rate but customer complaints increase. The trace dashboard reveals a new policy document version produces more reviewer rejections. Because events link workflow, model, knowledge, evaluator, and human outcome versions, the team rolls back the knowledge release and replays a synthetic evaluation set.
Raw customer messages are not present in general logs. Authorized support staff can retrieve a specific conversation through the source system under its access policy. The audit proves what the agent saw through references and approved snapshots.
Failure Cases to Diagnose
- Plain-text logs concatenate prompts: use structured, redacted events.
- Success means HTTP 200: include business evaluator and committed effect.
- No version fields: changes cannot be attributed.
- Average latency hides stalled tail: report percentiles and oldest age.
- Alerts have no owner: attach runbook and duty role.
- Logs editable by the agent: isolate write-only event collection and protect integrity.
- Retention is indefinite: set purpose-based periods and deletion.
🇵🇰 Pakistan Angle
Avoid placing phone numbers, CNIC values, bank references, CV contents, or WhatsApp transcripts into broad analytics dashboards. Use internal case references and grant detailed access only to staff with a business need.
Monitor provider and connectivity failures separately from model quality. A load-shedding-related worker outage, international API rate limit, and policy rejection need different responses. Show operating times in PKT to the team while keeping machine timestamps consistent.
Hands-On Exercise
- Define identifiers and ten structured events.
- Create a redaction and retention table.
- Build a run trace from trigger to committed effect.
- Define five service signals and two actionable alerts.
- Test secret leakage, version rollback, and log-access audit.
Done means: an operator can explain and recover a failed run without exposing sensitive data or trusting model prose.
Completion Rubric
- Runs, tasks, tools, approvals, and effects correlate.
- Events include policy/model/workflow versions.
- Redaction occurs before general logging.
- Percentiles, queues, failures, and cost are visible.
- Alerts have tested owners and runbooks.
- Log integrity, access, retention, and deletion are governed.
Sources
Key takeaway: log structured decisions and effects with correlation and versions, while minimizing sensitive content; observability must support recovery, not surveillance.