Module 08 · Database and Monitoring — Audit Logging and Model Evaluation
Evaluation Dashboard — Calibration, Drawdown, and Data Quality
Open lesson + course map
On this lesson
Course outline
Module 1 · Market Systems and Safety — Pehle Boundaries Samjho
Module 2 · Python Bot Architecture — Ek Professional Bot Ka Skeleton
Module 3 · Market Data Pipeline — Read-Only Evidence Safely Fetch Karo
Module 4 · AI Research Engine — Extraction Se Human Review Tak
Module 5 · Strategy Research — Hypothesis Se Paper Test Tak
Module 6 · Paper Execution Engine — Synthetic Fills Only
Module 7 · Risk Controls — Estimation Error and Paper Limits
Module 8 · Database and Monitoring — Audit Logging and Model Evaluation
Module 9 · Deploying the Research Service — Read-Only and Measured
The evaluation dashboard must prevent a smooth paper curve from hiding weak evidence. Lead with dataset coverage, freshness, exclusions, unresolved labels, and failed validations. Performance-like scenarios come later and remain in fictional points.
Panel one reconciles the denominator: discovered, captured, normalized, eligible, human-approved, paper-filled, resolved, pending, and excluded by reason. Panel two shows data quality by time/topic: missing fields, quote age, spread, rule changes, source conflicts, and outage gaps.
Panel three evaluates forecasts using Brier score, reliability bins, counts per bin, and a baseline comparison. Avoid tiny bins and show uncertainty. Panel four shows synthetic point path, maximum drawdown, cluster stress, blocked exits, and sensitivity to latency/slippage. It must say that paper fills are assumptions, not obtainable transactions.
Panel five covers operations: job success, retries, cache age, reconciliation, schema versions, alert incidents, and cost/storage budgets. Every chart links to the run manifest and configuration hash. Filters cannot change the primary preregistered result; exploratory views are labelled.
Accessibility matters: data tables accompany charts, colors are not the only signal, axes start and units are honest, and tooltips are keyboard accessible. Generated-at and data-through times are separate. If the newest run failed, the banner says so rather than presenting an old report as current.
Prevent dashboard cherry-picking. The default route loads the frozen primary evaluation; query parameters may expose labelled exploratory slices but cannot replace the headline. Every filtered view shows remaining rows, excluded rows, event-family count, and a shareable configuration hash. Empty or tiny slices render an insufficiency notice rather than a chart line.
Add snapshot regression tests for semantic content, not pixels alone: required safety banner, dataset version, denominator equation, baseline, uncertainty, stale state, and report timestamp. Parse generated HTML and assert links point to existing local manifests. A visual review at desktop and 390-pixel width checks overflow, table scrolling, keyboard focus, and right-to-left source excerpts.
// pakistan_angle
Pakistan Angle
Generate static, compressed HTML suitable for a modest laptop and limited data. Display Pakistan time as a view while retaining UTC. Include a plain-language Roman Urdu summary of data health, but keep technical metric definitions precise.
// hands_on
Hands-On Exercise
Build the five panels from a sealed fixture. Introduce a coverage gap, stale run, and concentrated theme; verify all three appear above the paper curve. Export accessible tables and ask a reviewer to trace one metric to its manifest.
// completion_rubric
Completion Rubric
5 checks — tick as you verify
// sources
Sources
3 official sources — check every claim yourself