Alerts exist to restore system health. They report stale data, failed validation, scheduler misfires, database errors, expired reviews, reconciliation mismatches, and cost-budget breaches. They never describe a paper direction, market opportunity, or desired outcome.
Define severity by operational impact: INFO for recovery, WARNING for degraded but safe behavior, ERROR for a failed run, and CRITICAL for corrupted provenance or safety-gate failure. Each alert includes issue key, first/last seen times, affected run/artifact IDs, sanitized error category, owner action, and runbook link.
Deduplicate by issue key and notification window. A repeated network failure updates count and last-seen instead of sending hundreds of emails. Escalate after a documented duration or consecutive failures. Send a recovery message once the health check passes, then close the incident record.
Use local structured logs as the reliable base. Optional SMTP sends minimal metadata and a dashboard link; no raw documents, prompts, secrets, or personal data belong in email. If SMTP fails, append ALERT_DELIVERY_FAILED locally. Alert failure must not hide the original incident.
Guard language with tests that reject terms such as buy, sell, bet, profit, winning, and opportunity in notification templates. This is not sufficient alone, so alerts also accept typed operational objects rather than free-form model text. An LLM never writes production alerts.
Track signal quality: incidents detected, duplicates suppressed, false alarms, time to acknowledgment, time to recovery, and alerts with missing runbooks. Tune from evidence without suppressing safety failures.
Test alert routing through a deterministic policy table rather than scattered conditionals. For example, provenance corruption is always CRITICAL and local-first; transient capture failure becomes WARNING until its consecutive-failure threshold; stale public report becomes ERROR once the freshness objective expires. Store the policy version with each incident so later tuning does not rewrite history.
Run a quarterly-style drill in the capstone: disable email, fill the disk threshold, and corrupt a projection copy. Verify the incident timeline preserves the primary fault, secondary delivery failure, escalation, operator notes, recovery check, and closure. Measure whether the runbook alone was sufficient for a new reviewer.
🇵🇰 Pakistan Angle
Email delivery and connectivity may be intermittent. Keep the local incident timeline authoritative and allow a lightweight digest. Do not send market content through WhatsApp or SMS under the label of monitoring; the course supplies no signals.
Hands-On Exercise
Create five typed incidents and route them through deduplication, escalation, recovery, and SMTP-failure fixtures. Verify secrets are redacted and banned action language fails tests. Produce one runbook for stale data with diagnose, contain, recover, and verify steps.
Completion Rubric
- Alerts are typed operational failures with clear owners/runbooks.
- Deduplication, escalation, recovery, and delivery failure are tested.
- Sensitive payloads and secrets cannot enter notifications.
- No alert contains action or profit language.
- Local incident history remains authoritative.
Sources
Key takeaway: Good alerts shorten failure recovery without turning an engineering system into a signal channel.