Metrics should reveal where a customer journey fails. Averages, message counts, and chatbot containment can hide abandoned users and overloaded agents. Define every event, denominator, time boundary, and exclusion before reporting a rate.
After this lesson, you can build an operational scorecard that balances commercial outcomes, service quality, consent health, and system reliability.
Create an Event Dictionary
For each event specify producer, timestamp, identifiers, and qualification:
| Event | Meaning | Authoritative producer |
|---|---|---|
conversation_started | eligible inbound or permitted outbound thread begins | messaging platform |
lead_qualified | versioned rule completes | qualification service |
handoff_created | human case enters queue | inbox/workflow |
agent_acknowledged | named owner accepts | inbox |
order_confirmed | order system commits | commerce system |
payment_settled | trusted verification succeeds | payment service |
opted_out | recipient requests stop | consent ledger |
case_resolved | outcome and closure recorded | service system |
Use one correlation ID to trace the journey without making the phone number the analytics key.
Calculate Named Metrics
First acknowledgement time = acknowledged_at - handoff_created_at
First substantive response = first_helpful_agent_reply - handoff_created_at
Qualification rate = qualified leads / eligible conversations
Qualified-to-order rate = confirmed orders / qualified leads
Payment completion rate = settled payments / payment attempts
Opt-out rate = opt-outs / delivered campaign messages (when delivery is known)
Automation recovery rate = recovered fallbacks / fallback events
Report sample size and period. Use median and 90th percentile for response time. A 2-minute average can coexist with a group waiting six hours.
Segment Carefully
Useful operational segments include source campaign, flow version, product, service team, business-hours status, and failure reason. Avoid profiling by sensitive traits or proxies. Small segments can expose individual behavior; apply access controls and minimum reporting sizes.
Compare before/after only with caveats. Traffic, offer, season, staffing, and price may have changed. A controlled experiment needs an ethical assignment rule, stable definitions, sufficient sample, and a predeclared outcome.
Worked Example
During a sample week, a service business records 500 eligible conversations, 300 qualified leads, 240 handoffs, 60 confirmed orders, and 45 settled payments.
Qualification rate = 300 / 500 = 60%
Qualified-to-order rate = 60 / 300 = 20%
Payment completion rate = 45 / 60 = 75%
Median acknowledgement is 8 minutes and the 90th percentile is 52 minutes. After-hours cases explain most long waits, while one flow version produces twice the fallback share of another. The action is to repair that flow and show honest after-hours expectations—not to celebrate the 8-minute median alone. These figures are examples, not benchmarks.
Failure Cases to Diagnose
- Messages sent used as the conversion denominator: choose eligible conversations or qualified leads explicitly.
- Chatbot containment rewarded at all costs: include repeat contacts, fallback, complaint, and resolution.
- Average response time only: report percentiles and business-hours segments.
- Payment inferred from order: use the payment service event.
- Attribution claims causality: label observational comparisons honestly.
- Dashboard stores raw private messages: aggregate and restrict access.
- Metric definition changes silently: version the event dictionary.
🇵🇰 Pakistan Angle
Separate business-hours and after-hours performance in PKT. Ramadan, Eid, sales events, weather, courier disruption, and load shedding can shift demand and staffing; annotate these periods instead of presenting a false trend. Compare cities only when service rules and sample sizes are comparable.
Track COD confirmation, dispatch, delivery, refusal, and return separately if relevant. Do not label every refusal as fraud. Diagnose wrong item, unexpected fee, late delivery, duplicate order, address issue, or changed mind using documented outcome codes.
Hands-On Exercise
- Define eight core events and authoritative producers.
- Write formulas with named denominators.
- Build a weekly scorecard with sample sizes, median, and p90.
- Add consent-health and reliability metrics.
- Review one apparent improvement for alternative explanations.
Done means: another analyst can reproduce every number and trace a change to a flow, queue, offer, or system condition without reading private conversations.
Completion Rubric
- Event definitions and producers are versioned.
- Every rate has an explicit denominator and period.
- Response time includes median and p90.
- Financial outcomes use authoritative events.
- Consent and reliability sit beside revenue metrics.
- Segmentation protects privacy and avoids sensitive inference.
Sources
- WATI Help Center — Analysis and Reporting documentation
- WATI Help — Team Inbox
- NIST — Privacy Framework
Key takeaway: make metrics reproducible from governed events and honest denominators, then use percentiles, negative signals, and failure reasons to improve the system.