The evaluation dashboard must prevent a smooth paper curve from hiding weak evidence. Lead with dataset coverage, freshness, exclusions, unresolved labels, and failed validations. Performance-like scenarios come later and remain in fictional points.
Panel one reconciles the denominator: discovered, captured, normalized, eligible, human-approved, paper-filled, resolved, pending, and excluded by reason. Panel two shows data quality by time/topic: missing fields, quote age, spread, rule changes, source conflicts, and outage gaps.
Panel three evaluates forecasts using Brier score, reliability bins, counts per bin, and a baseline comparison. Avoid tiny bins and show uncertainty. Panel four shows synthetic point path, maximum drawdown, cluster stress, blocked exits, and sensitivity to latency/slippage. It must say that paper fills are assumptions, not obtainable transactions.
Panel five covers operations: job success, retries, cache age, reconciliation, schema versions, alert incidents, and cost/storage budgets. Every chart links to the run manifest and configuration hash. Filters cannot change the primary preregistered result; exploratory views are labelled.
Accessibility matters: data tables accompany charts, colors are not the only signal, axes start and units are honest, and tooltips are keyboard accessible. Generated-at and data-through times are separate. If the newest run failed, the banner says so rather than presenting an old report as current.
Prevent dashboard cherry-picking. The default route loads the frozen primary evaluation; query parameters may expose labelled exploratory slices but cannot replace the headline. Every filtered view shows remaining rows, excluded rows, event-family count, and a shareable configuration hash. Empty or tiny slices render an insufficiency notice rather than a chart line.
Add snapshot regression tests for semantic content, not pixels alone: required safety banner, dataset version, denominator equation, baseline, uncertainty, stale state, and report timestamp. Parse generated HTML and assert links point to existing local manifests. A visual review at desktop and 390-pixel width checks overflow, table scrolling, keyboard focus, and right-to-left source excerpts.
🇵🇰 Pakistan Angle
Generate static, compressed HTML suitable for a modest laptop and limited data. Display Pakistan time as a view while retaining UTC. Include a plain-language Roman Urdu summary of data health, but keep technical metric definitions precise.
Hands-On Exercise
Build the five panels from a sealed fixture. Introduce a coverage gap, stale run, and concentrated theme; verify all three appear above the paper curve. Export accessible tables and ask a reviewer to trace one metric to its manifest.
Completion Rubric
- Denominators and data quality precede scenario results.
- Calibration includes counts, uncertainty, and baseline.
- Drawdown and costs remain fictional/scenario-labelled.
- Every panel traces to versions and hashes.
- Stale/failed runs are impossible to mistake as current.
Sources
Key takeaway: An honest dashboard foregrounds evidence quality, uncertainty, complete denominators, and measured freshness before showing any synthetic result.