Choose by evidence, not rankings
Model selection is task-specific and date-specific. Select a service, model, configuration, and workflow by testing the work you actually need under your real controls.
Capabilities, model names, plans, prices, regions, integrations, and data terms change. Check current official documentation and procurement terms. Test with synthetic or approved redacted samples; never “compare models” by uploading live board, employee, customer, contract, or credential data.
Any production implementation additionally requires approved redacted inputs; fact, inference, and proposal labels; rights and lawful basis; data minimization; least privilege; tenant separation; managed secrets; audit logs; representative, adversarial, and regression evaluations; proposal-only behavior; human approval; monitoring; a stop condition; an incident path; and tested rollback.
Start with a task specification
Build a representative evaluation set
Use the same versioned cases and rubric for every candidate. Include ordinary, difficult, and failure cases rather than one impressive demonstration.
- Representative Arabic and English examples from permitted, synthetic data
- Dialect, formatting, accessibility, and domain terminology checks relevant to users
- Adversarial instructions, malicious documents, ambiguous requests, and missing evidence
- Expected abstention when a claim cannot be supported
- Regression cases rerun after model, prompt, retrieval, policy, or integration changes
A fictional commerce team evaluates two candidate configurations for a public-source weekly brief. Reviewers are blinded to provider names. One configuration writes more fluently; the other preserves citations better. Because citation fidelity is a launch gate, fluency alone does not determine selection. The result applies only to that test set, version, language mix, and date.
Score the whole system
Quality
Human-reviewed correctness, source traceability, useful uncertainty, language performance, and task completion.
MEASUREControl
Access, retention, training terms, data location, identity, logs, tenant boundaries, secret handling, and deletion.
VERIFYOperations
Observed reliability, recoverability, support, total workflow cost, portability, and change management.
OWNDo not infer production fitness from a leaderboard, vendor demo, one prompt, or a personal subscription. Do not bypass enterprise controls for “better output.” A selected configuration is a dated hypothesis that must be re-evaluated when the system or task changes.
Exercise: a small, reviewable bake-off
- Choose one low-impact task and write its pass/fail requirements.
- Create synthetic cases with expected evidence and safe failure behavior.
- Have independent reviewers score anonymized outputs against the same rubric.
- Record model and configuration versions, date, limitations, cost basis, and disagreements.
- Keep the winner in proposal-only mode; re-run the suite before expanding access or use.
Completion gate
A defensible selection record includes
Consult the providers’ current documentation before evaluating: OpenAI model selection guidance, Anthropic model overview, and Google Gemini model documentation. Their documentation does not replace your own task-specific evaluation.