ENع
PHASE 3 — DAY 10

Readiness Review, Pilot, Improve

A working demo is the beginning of evidence collection. Today you decide whether the workflow should stop, be revised, or enter a separately authorized pilot.

REVIEW WORKSHOPGOVERNANCEFINAL DAY

This course does not grant production authorization. Operational authority comes from the organization that owns the systems, data, risk, and affected relationships. Security, privacy, legal, accessibility, domain, and operational owners must review the actual implementation and use case.

Part 1: Run the Readiness Gate

1
Purpose and owner — approved use case, prohibited uses, accountable owner, users, and affected people.
2
Data — lawful basis, minimization, consent where required, retention, deletion, residency, and vendor terms.
3
Security — least privilege, secret management, tenant isolation, dependency review, threat model, and abuse tests.
4
Evaluation — representative and adversarial cases, predefined thresholds, baseline comparison, and known limitations.
5
Human control — authenticated approval before consequences, accessible appeal, and no model self-approval.
6
Operations — cost and volume caps, monitoring, alert owners, availability needs, and vendor-failure behavior.
7
Recovery — tested stop control, rollback, incident classification, evidence preservation, communication, and recovery criteria.
8
Change management — training, disclosure, versioning, re-evaluation triggers, and documented approval.

A failed or missing gate means do not progress. Record the gap, owner, and required evidence. A risk register is not a waiver.

Part 2: Use a Graduated Evidence Ladder

Progress only with authorization and exit criteria
  1. Local synthetic test: isolated files, no external effects.
  2. Service sandbox: test accounts and non-production credentials with hard caps.
  3. Shadow: compare draft output with the current process; the workflow cannot act.
  4. Canary: a small, approved scope with close monitoring and an immediate kill switch.
  5. Guarded operation: only if all owners approve, thresholds hold, and rollback remains tested.

Define entry criteria, exit criteria, maximum exposure, monitoring, stop triggers, and rollback for each stage. “It worked in the demo” is not an exit criterion.

Part 3: Choose Models With Task-Specific Evals

There is no permanent best-model ranking. Models, versions, prices, limits, and behavior change. For each task, compare eligible candidates on the same representative and adversarial eval set. Measure the quality criteria that matter, plus latency, cost, tool behavior, safety, privacy requirements, and operational fit. Pin versions where supported and rerun regression tests before a change.

Source review: this guidance was reviewed 31 August 2026 against OpenAI’s current model-selection guidance. Also verify the current official documentation for every provider you evaluate.

Part 4: Measure Claims, Do Not Assume Them

Write a hypothesis and a baseline before the pilot. Examples: review acceptance rate, correction rate, task completion time, cost per accepted draft, policy-violation rate, or accessibility defects. Define the unit, sample, measurement window, exclusions, and decision threshold for this use case. Report uncertainty and adverse effects.

A
Claim — a testable hypothesis, not a promised saving or outcome.
B
Method — baseline, sample, rubric, reviewer, and comparison.
C
Result — observed values, uncertainty, failures, and subgroup effects where relevant.
D
Decision — stop, revise, continue the pilot, or seek broader approval.

Part 5: Plan for People and Accountability

Automation changes tasks, review load, skills, incentives, and error ownership. Job security, productivity, income, and headcount outcomes remain uncertain and context-specific. Consult affected teams, worker representatives where applicable, accessibility experts, legal and compliance owners, and people represented in the data. Explain where AI is used, provide training, and create a meaningful route to question or appeal consequential decisions.

Part 6: Rehearse an Incident

Run a tabletop exercise; do not connect to production. Scenario: monitoring detects that the workflow used an unauthorized destination and may have exposed restricted data. Produce a timed decision log covering: - immediate containment and credential revocation; - evidence preservation without copying sensitive payloads unnecessarily; - affected-system and data-scope investigation; - legal, privacy, security, vendor, and customer communication owners; - rollback or clean recovery criteria; - post-incident fixes, regression tests, and reauthorization. Identify missing contacts, controls, logs, or runbooks. Do not claim recovery until the owner verifies each criterion.

Part 7: Maintain an Evidence-Based Operating Cadence

At a cadence appropriate to the risk, review evaluation drift, incidents, near misses, dependency and model changes, access, spend, user feedback, and unresolved limitations. Any material change to data, model, prompt, tool, permission, destination, or decision scope triggers proportionate regression testing and reapproval.

Final Checklist

Evidence required; course completion is not authorization
← Day 9: Reliable OrchestrationFramework Review Complete

Keep the Evidence, Not the Hype

Revisit earlier lessons, improve the lab, and ask the appropriate owners for review before expanding its scope.

Review the Roadmap