PART 03 · FLAGSHIP

AI × Customer Service

How to test ticket assistance, voice, routing, self-service, quality review, and WhatsApp without hiding uncertainty or removing consent, accessibility, and human escalation.

Representative evaluation · human escalation · bounded, reversible actions

Chatbots & Virtual Agents

Begin with retrieval and draft assistance for a narrow, low-risk task; expand only when reviewed evidence supports it.

BASELINE RISK

Fragmented support context

Policies, order data, and prior contact are spread across tools → repeated work and inconsistent answers are hard to measure → accessibility and escalation vary.

CONTROLLED PILOT

Grounded draft with safe handoff

Retrieve from approved, versioned sources → cite the evidence → draft an accessible response → disclose automation where appropriate → route uncertainty, complaints, distress, exceptions, and human requests → require approval for refunds, credits, cancellations, or account changes.

Baseline worksheet: document current ticket count, handling-time assumption, and loaded hourly cost. It does not estimate automation or savings.

Evidence to collect

CORRECT
Resolved to policy
Sample complete conversations and verify facts, citations, policy version, task completion, and reopened contacts—not the bot's self-reported “resolution.”
Evidence: human-reviewed outcome set
ESCALATE
Safety recall
Test complaints, chargebacks, threats, distress, vulnerable users, regulated topics, policy exceptions, low confidence, and direct requests for a person.
Evidence: adversarial escalation set
ACCESS
Inclusive completion
Evaluate screen-reader flow, keyboard access, plain language, Arabic dialects, code-switching, spelling errors, low bandwidth, and alternative contact paths.
Evidence: accessibility review
CONTROL
Consequential actions
Keep refunds, credits, cancellations, and account changes PAUSED; require least privilege, signed exact approval, caps, idempotency, logs, stop, and rollback.
Evidence: policy and failure tests
Vendor resolution claims use different task mixes, definitions, exclusions, and review methods. Verify precise dated methodology and reproduce results on your representative, adversarial support set.

Voice Agents

Voice assistance needs explicit disclosure, recording consent where required, an accessible alternative, and immediate human escalation.

BASELINE RISK

Notice, consent, identity checks, language support, recording retention, and escalation are inconsistent → sensitive content can be captured unnecessarily.

CONTROLLED PILOT

Disclose the automated system → obtain required recording consent → offer keypad/text/human alternatives → authenticate before account data → minimise/redact recordings → escalate uncertainty or sensitive intent → keep bookings, payments, and account changes as approved drafts.

Pilot evidence to collect

HEAR
Representative speech
Test accents, dialects, code-switching, names, numbers, noise, pauses, speech disabilities, and low-quality connections.
Evidence: consented reviewed test set
SAFE
Escalation and identity
Measure failed authentication, sensitive disclosure, distress, collection/medical/legal topics, interruption, and human-handoff reliability.
Evidence: red-team call log
LAWFUL
Contact and recording
Verify outreach eligibility, do-not-call/suppression, disclosure, recording consent, retention, deletion, and vendor subprocessors by jurisdiction.
Evidence: approved compliance record
Do not use voice AI for unsolicited bulk calling or infer that a vendor demonstration is production evidence. Verify current telephony, consent, recording, accessibility, and sector rules.

Ticket Routing & Prioritisation

Routing can assist triage, but urgency, language, sentiment, and intent labels require evaluation and a safe manual path.

BASELINE RISK

Queue rules and categories are undocumented → reroutes and missed urgency are not measured → labels can encode subjective or biased assumptions.

CONTROLLED PILOT

Define routing taxonomy and service capacity → test with representative languages and groups → use confidence thresholds → default uncertainty to manual triage → preserve override → monitor missed urgency, reroutes, fairness, and SLA impact.

Pilot evidence to collect

ROUTE
End-to-end correctness
Measure final destination, reroutes, handling delay, capacity mismatch, and whether the assigned team could actually resolve the issue.
Evidence: sampled route audit
URGENT
Cost-sensitive recall
Prioritise detection of safety, fraud, vulnerable customers, legal deadlines, and serious complaints; false negatives may matter more than aggregate accuracy.
Evidence: adversarial urgency set
FAIR
Language and group review
Compare routing and wait outcomes by language, dialect, accessibility need, region, and relevant customer groups; investigate proxy effects.
Evidence: fairness and override report
A classification accuracy claim does not prove good routing. Verify the label set, class balance, evaluation data, threshold, capacity constraints, and business outcome.

Self-Service

Self-service is useful when it gives a correct, accessible answer and a clear path to a person—not when it merely suppresses contact.

BASELINE RISK

Policies are stale or difficult to navigate → search does not match customer language → contact paths are hidden → repeat contacts are mislabelled as deflection.

CONTROLLED PILOT

Version approved sources → retrieve and cite the applicable passage → show effective date and uncertainty → use accessible interaction patterns → offer human contact at every step → measure verified completion and repeat contact.

Pilot evidence to collect

COMPLETE
Verified task outcome
Confirm the customer reached the right outcome, did not return for the same issue, and could still contact a person.
Evidence: sampled journey review
CURRENT
Source freshness
Track content owner, effective date, jurisdiction, product/version applicability, review date, and withdrawal of superseded guidance.
Evidence: knowledge-source register
“Deflection” can hide abandonment or blocked access. Verify task completion, repeat contacts, complaints, accessibility, and human-contact availability.

Quality Assurance

Automated review can broaden screening, but model scores are not ground truth and must not silently determine employee discipline.

BASELINE RISK

Sampling and rubric choices are undocumented → rare high-severity failures may be missed → employees cannot understand or contest conclusions.

CONTROLLED PILOT

Define a task-specific rubric with staff input → stratify sampling by risk → validate model findings against trained reviewers → test languages and groups → expose evidence and uncertainty → require human review and an appeal path before employment consequences.

Pilot evidence to collect

AGREE
Human-model agreement
Measure agreement and disagreement by rubric item, severity, language, channel, and reviewer; adjudicate a representative sample.
Evidence: blinded validation set
RARE
High-severity recall
Deliberately test privacy disclosure, discrimination, threats, fraud, vulnerable customers, regulated advice, and missed escalation.
Evidence: adversarial QA set
FAIR
Employment safeguards
Review error patterns across employees, languages, accents, schedules, and customer mixes; provide evidence, human review, and contestability.
Evidence: impact and appeal audit
Screening more conversations is not equivalent to correctly judging them. Verify recording/monitoring obligations, rubric validity, reviewer agreement, fairness, and employee appeal requirements.

WhatsApp Business

WhatsApp can be one consented support channel. Design for current platform rules, accessible alternatives, and reliable human handoff.

BASELINE RISK

Channel eligibility, template rules, consent, identity, accessibility, and handoff are handled inconsistently → messages can expose account information or persist too long.

CONTROLLED PILOT

Confirm opt-in and conversation window → disclose assistance → authenticate before account data → retrieve approved sources → minimise data → provide accessible alternatives → hand off uncertainty and human requests → approve consequential actions separately.

Baseline worksheet: document current monthly inquiries and reported after-hours share. It does not estimate resolution or repeat purchase.

Pilot evidence to collect

ELIGIBLE
Consent and window
Test opt-in provenance, template eligibility, suppression, conversation-window enforcement, opt-out, and preference updates.
Evidence: channel eligibility log
RESOLVE
Verified customer outcome
Review complete journeys, policy correctness, repeat contact, abandonment, complaints, accessibility, and whether human help was reachable.
Evidence: sampled journey review
WRITE
Bounded actions
Keep orders, returns, refunds, credits, bookings, and account edits PAUSED; require signed exact approval, least privilege, caps, idempotency, logs, stop, and rollback.
Evidence: control and failure tests
Vendor examples cannot establish a regional resolution benchmark. Verify current official channel documentation and measure your own task mix with a stable definition of correct resolution.
MYTH
"There is one honest automation percentage for customer service."
Tap to see reality →
REALITY
Results depend on task mix, policy quality, language, channel, definition, review window, escalation design, and risk tolerance. Publish a task-level evaluation and never optimise “deflection” by making human help harder to reach.

Reference pattern: a controlled support workflow.

Start with read-only retrieval and drafts, evaluated on representative and adversarial conversations. Separate tenants; minimise and encrypt data; use least-privilege identities; enforce consent, suppression, identity checks, accessibility, retention, redaction, and human escalation outside the model. Refunds, credits, returns, bookings, messages, and account changes stay PAUSED until deterministic policy validates an unexpired signed approval bound to the exact customer, resource, values, and time window. Require caps, idempotency, complete logs, stop controls, and tested rollback.

ZendeskFreshdeskWhatsApp APIShopifyIntercom
Educational architecture only. Production readiness requires local legal, privacy, security, accessibility, fairness, reliability, and channel-policy review.

Vendor discovery links — verify before use

These links identify products mentioned in the earlier lesson. They do not prove a performance result. Locate precise, dated official documentation, terms, security/privacy material, and case methodology before relying on a feature or claim.

  1. Intercom Fin 2 — intercom.com
  2. Klarna AI — klarna.com
  3. Zendesk — zendesk.com
  4. Ada.cx — ada.cx
  5. Bland AI — bland.ai
  6. Retell AI — retell.ai
  7. Vapi — vapi.ai
  8. Zendesk Triage — zendesk.com
  9. Forethought Solve — forethought.ai
  10. SentiSum — sentisum.com
  11. Stonly — stonly.com
  12. Shopify AI — shopify.com
  13. Klaus / Zendesk QA — klausapp.com
  14. Observe.AI — observe.ai
  15. CallMiner — callminer.com
  16. Haptik — haptik.ai
  17. WATI — wati.io
  18. Zoko — zoko.io