عربي
DAY 02 · EVALUATION

Choose by evidence, not rankings

Model selection is task-specific and date-specific. Select a service, model, configuration, and workflow by testing the work you actually need under your real controls.

SELF-PACEDTEST HARNESSREVIEWED 31 AUG 2026

Capabilities, model names, plans, prices, regions, integrations, and data terms change. Check current official documentation and procurement terms. Test with synthetic or approved redacted samples; never “compare models” by uploading live board, employee, customer, contract, or credential data.

Any production implementation additionally requires approved redacted inputs; fact, inference, and proposal labels; rights and lawful basis; data minimization; least privilege; tenant separation; managed secrets; audit logs; representative, adversarial, and regression evaluations; proposal-only behavior; human approval; monitoring; a stop condition; an incident path; and tested rollback.

Start with a task specification

1Decision: what will the output inform, and who remains accountable?
2Inputs: approved data classes, languages, formats, rights, retention, and residency requirements.
3Quality: citation fidelity, completeness, calibrated uncertainty, instruction following, and accessibility.
4Risk: privacy, prompt injection, unsafe advice, bias, data leakage, and over-confident fabrication.
5Operations: latency and cost measured on your workload, support, portability, logs, access, and exit plan.

Build a representative evaluation set

Use the same versioned cases and rubric for every candidate. Include ordinary, difficult, and failure cases rather than one impressive demonstration.

A fictional commerce team evaluates two candidate configurations for a public-source weekly brief. Reviewers are blinded to provider names. One configuration writes more fluently; the other preserves citations better. Because citation fidelity is a launch gate, fluency alone does not determine selection. The result applies only to that test set, version, language mix, and date.

Score the whole system

Quality

Human-reviewed correctness, source traceability, useful uncertainty, language performance, and task completion.

MEASURE
Control

Access, retention, training terms, data location, identity, logs, tenant boundaries, secret handling, and deletion.

VERIFY
Operations

Observed reliability, recoverability, support, total workflow cost, portability, and change management.

OWN

Do not infer production fitness from a leaderboard, vendor demo, one prompt, or a personal subscription. Do not bypass enterprise controls for “better output.” A selected configuration is a dated hypothesis that must be re-evaluated when the system or task changes.

Exercise: a small, reviewable bake-off

  1. Choose one low-impact task and write its pass/fail requirements.
  2. Create synthetic cases with expected evidence and safe failure behavior.
  3. Have independent reviewers score anonymized outputs against the same rubric.
  4. Record model and configuration versions, date, limitations, cost basis, and disagreements.
  5. Keep the winner in proposal-only mode; re-run the suite before expanding access or use.

Completion gate

A defensible selection record includes

Consult the providers’ current documentation before evaluating: OpenAI model selection guidance, Anthropic model overview, and Google Gemini model documentation. Their documentation does not replace your own task-specific evaluation.

← Day 1: AccountabilityDay 3: Evidence-first briefs →