Run the same bounded task through candidate tools, score what you observe, and document why a tool is fit for this use case — without turning today’s result into a permanent ranking.
GUIDED EVALUATION LABSYNTHETIC DATAREVIEWED 31 AUG 2026
The safe outcome is an evaluation record, not three new accounts or a universal winner. Products, models, limits, prices, interfaces, and data terms change. Check each provider’s current documentation on the day you test. All exercises here use fictional or synthetic material; every output remains a draft or proposal until an accountable person reviews it.
Data boundary: do not paste customer, competitor, employee, client, credential, contract, or confidential business data into a consumer chat. Before any later authorized use, confirm rights or lawful basis, minimize fields, review vendor terms and retention, configure access, and follow your organization’s approval process.
Part 1: Define the Decision Before Testing
01
Use case: one narrow job, user, and decision. Example: “Draft a fictional weekly campaign summary for a marketing analyst to review.”
02
Allowed input: synthetic rows and public facts you are permitted to use. List prohibited data explicitly.
03
Expected output: a defined schema, citation rules, uncertainty labels, and a refusal or escalation response for unsupported requests.
04
Decision owner: name who reviews the draft, what they verify, and what the tool is never allowed to publish, send, upload, or change.
Start with requirements. A model that performs well on one writing task may fail a multilingual, structured, visual, or citation-heavy task. Provider benchmark claims are useful context, but they do not replace tests on your work.
Part 2: Build One Reusable Evaluation Set
Create a small test pack that every candidate receives unchanged. Keep the prompt, source packet, settings, and scoring rubric together so another reviewer can reproduce the comparison.
Representative
Normal synthetic cases covering the languages, formats, lengths, and decisions the task really needs.
Question: useful on ordinary work?
Boundary
Empty values, conflicting sources, near-threshold numbers, missing dates, and requests just outside scope.
Question: safe at the edges?
Malformed & adversarial
Broken schemas, prompt injection inside a document, fabricated citations, secrets, and instructions to bypass policy.
Question: fail closed or improvise?
Predefine the scoring rubric
Grounded accuracy: every factual statement is supported by the supplied source packet; calculations are independently recomputed.
Completeness: required fields are present, without inventing missing information.
Format validity: output passes the schema or template exactly.
Uncertainty and refusal: unknowns are visible; prohibited or unsupported requests stop and escalate.
Language and accessibility: Arabic and English meaning, tone, reading order, and labels work for the intended audience.
Operational fit: observed latency, cost per completed test, availability, administration, retention controls, and reviewer effort fit the use case.
Use case ID: [ID]
Review date: [DATE]
Candidate and version shown by provider: [VALUE]
Interface or API surface: [VALUE]
Settings and tools enabled: [VALUE]
Test-case ID: [REP / BOUNDARY / MALFORMED / ADVERSARIAL]
Observed output: [STORE ONLY APPROVED SYNTHETIC OUTPUT]
Criterion scores and evidence: [TABLE]
Failure or refusal behavior: [OBSERVED]
Latency / usage / reviewer effort: [OBSERVED]
Decision: [PASS / REVISE / REJECT]
Reviewer and rationale: [NAME + NOTES]
Part 3: Run a Fair Candidate Comparison
Read the current provider documentation and record the exact candidate/version presented to you. Do not copy model names from this course.
Disable unnecessary connectors, memory, browsing, file access, and actions. Use a separate test workspace with no production credentials.
Run the same frozen cases and prompt contract. Do not quietly give one candidate extra hints.
Score observable outputs against the predefined rubric. Keep facts, reviewer judgment, and provider claims in separate columns.
Repeat enough cases to expose inconsistency. Investigate failures instead of averaging a dangerous miss away.
Select the smallest acceptable option for this use case, or select none. Record limitations, retest triggers, and an expiry date for the decision.
NO PERMANENT LEADERBOARD
A comparison is valid only for the tested task, data, language, settings, date, and acceptance criteria. A provider update, changed prompt, new source type, or material drift triggers regression testing on the same suite.
Part 4: Three Safe Conversation Patterns
A
Ask: request an explanation grounded only in named sources. Require citations, distinguish source fact from inference, and permit “insufficient evidence.”
C
Create: draft from a fictional brief with explicit audience, constraints, rights, output schema, prohibited claims, and reviewer.
Z
Analyze: use a synthetic dataset with a data dictionary. Ask for checks and calculations, then verify them outside the model before accepting a finding.
Using only the public sources listed below, create a dated comparison of their stated requirements. Cite the source next to each claim. Put unsupported questions in an “Unknown” section; do not infer a current fact from memory.
Draft a campaign brief for the fictional brand “Northstar Coffee.” Use only this invented brief. Return the specified headings, flag assumptions, make no performance promise, and stop if a required field is missing. This is a proposal for human review, not publishable copy.
Analyze this generated table of 12 fictional campaigns. Validate column types and totals first. If a value is missing or contradictory, list it instead of repairing it. Return calculations with formulas and a separate “possible interpretations” section. Recommend no live action.
Part 5: Read Output Like a Reviewer
01
Provenance: can you open every citation, and does it support the exact claim?
02
Evidence: which statements are supplied facts, calculations, assumptions, or model inferences?
03
Failure: did the model expose missing data, conflicts, and uncertainty, or hide them behind fluent language?
04
Impact: what could happen if this draft is wrong, and which qualified reviewer must approve it?
05
Boundary: confirm the output has not been sent, published, uploaded, or used to change a live system.