Module 04 · Operational Guardrails
Logging and Observability for Autonomous Workflows
Open lesson + course map
On this lesson
Course outline
Module 1 · Agent Fundamentals
Module 2 · Agent Frameworks
Module 3 · Swarm Orchestration
Module 4 · Operational Guardrails
Module 5 · Shipping a Real Agent System
Observability answers: what run happened, which version acted, what tools were requested, what effects committed, why it stopped, and how to recover. Logging every prompt and private record is not observability; it is often a data leak.
After this lesson, you can define structured events, trace a run, and redact sensitive content.
// concept
Use a Correlation Model
Generate run_id, task_id, trace_id, and stable effect/idempotency IDs. Emit structured events:
{
"event": "tool.completed",
"run_id": "run_72",
"task_id": "task_19",
"tool": "create_quote_draft",
"tool_version": "3",
"result": "success",
"duration_ms": 418,
"attempt": 1,
"effect_id": "quote:D-44",
"recorded_at": "2026-07-19T10:00:00Z"
}Record model/provider identifier, prompt/policy version, usage, evaluator version, approval reference, state transition, error class, and stop reason where relevant.
// concept
Redact at Collection
Do not log secrets, tokens, authorization headers, OTPs, passwords, full payment data, or unnecessary identity fields. Prefer references and hashes over raw phone numbers, emails, documents, and customer messages. Restrict access and retention, protect log integrity, and record administrative access.
Keep a separately controlled debug mode for approved synthetic environments. “We may need it later” is not a retention purpose.
// concept
Define Service Signals
Track:
- task success and evaluator pass rate;
- invalid or denied tool calls;
- model/tool latency percentiles;
- retry, timeout, dead-letter, and circuit state;
- queue depth and oldest task age;
- tokens/cost per completed outcome;
- human approval wait and rejection reasons;
- duplicate effect prevented;
- drift by workflow/model/version.
Alert on actionability, not noise. Every alert needs owner, threshold rationale, runbook, and test.
// worked_example
Worked Example
A support-draft agent shows a normal completion rate but customer complaints increase. The trace dashboard reveals a new policy document version produces more reviewer rejections. Because events link workflow, model, knowledge, evaluator, and human outcome versions, the team rolls back the knowledge release and replays a synthetic evaluation set.
Raw customer messages are not present in general logs. Authorized support staff can retrieve a specific conversation through the source system under its access policy. The audit proves what the agent saw through references and approved snapshots.
// failure_cases
Failure Cases to Diagnose
7 cases to diagnose
Plain-text logs concatenate prompts
use structured, redacted events.
Success means HTTP 200
include business evaluator and committed effect.
No version fields
changes cannot be attributed.
Average latency hides stalled tail
report percentiles and oldest age.
Alerts have no owner
attach runbook and duty role.
Logs editable by the agent
isolate write-only event collection and protect integrity.
Retention is indefinite
set purpose-based periods and deletion.
// pakistan_angle
Pakistan Angle
Avoid placing phone numbers, CNIC values, bank references, CV contents, or WhatsApp transcripts into broad analytics dashboards. Use internal case references and grant detailed access only to staff with a business need.
Monitor provider and connectivity failures separately from model quality. A load-shedding-related worker outage, international API rate limit, and policy rejection need different responses. Show operating times in PKT to the team while keeping machine timestamps consistent.
// hands_on
Hands-On Exercise
5 steps
Define identifiers and ten structured events.
Create a redaction and retention table.
Build a run trace from trigger to committed effect.
Define five service signals and two actionable alerts.
Test secret leakage, version rollback, and log-access audit.
// completion_rubric
Completion Rubric
6 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself
// check_yourself
Check yourself
4 questions · answers and options are taken word-for-word from this course
1 / 4 · diagnose
Your work shows this failure mode: “Plain-text logs concatenate prompts.” What does the lesson tell you to do about it?