PHASE 06 · TRAIN

Train Like a Manager — The Feedback Loop

A successful first run is not evidence that an assistant is dependable. Build a repeatable evaluation loop around the exact task, its failure modes, and the level of human review the risk requires.

Risk-based · review after each material test or change Iteration Compounding work

The mindset shift: training ≠ retraining the model

To be clear: you are not "training the AI" in the sense of changing the model's weights. You're training your assistant configuration — the system prompt, the SOP, the knowledge files, the examples library. That's where every improvement lives.

★ THE INSIGHT

The AI doesn't learn from your feedback in the moment. You learn from it. You then update the SOP / system prompt / knowledge files so that the next conversation gets the lesson baked in. Training = your discipline to capture and re-inject what you noticed.

Written feedback can improve the configuration, but an AI assistant is not a human hire and a note does not make a correction permanent. Treat every prompt, model, tool, knowledge, or policy change as a new version that must be evaluated again.

The training loop

The loop is simple, and it's the same one every great manager uses:

①Stage. Run a representative synthetic or specifically authorized, minimized pilot in an approved workspace; never provide secrets.
②Review. Read the output carefully. Compare to the quality bar.
③Diagnose. If something's off, ask: was the SOP missing this? Was the knowledge stale? Was the prompt unclear?
④Update the source. Edit the SOP / system prompt / knowledge file to prevent the same mistake next time.
⑤Re-run. Same task, fresh conversation, updated config. Verify the fix held.
⑥Log. Note what you changed and why in the assistant's feedback-log.md.

Each useful iteration makes the configuration more explicit and testable. It can also create a regression elsewhere, so preserve the previous version and re-run a representative test set before promotion.

The feedback file pattern

For every assistant, keep a feedback-log.md. Open it every time you correct an output. Three columns is enough:

# Maya · Caption Writer · Feedback Log | Date | What went wrong | What I changed | |------|-----------------|----------------| | 2026-05-03 | Used "elevate" in caption B despite hard-no rule. | Re-read system prompt — "elevate" was listed but list was buried. Moved hard-nos to top of prompt with caps. | | 2026-05-04 | All 3 variants used same opening pattern ("Two months of..."). | Added Operating Principle: "Each variant must use a meaningfully different opening structure." | | 2026-05-06 | Caption mentioned product price even though brand isn't price-sensitive. | Added to Conditions: "Never mention price unless brief explicitly says to." | | [TIMESTAMP + TZ] | Synthetic output used unsupported tone labels. | Added rights-cleared counterexamples, source/context/exception fields, and a reviewer rule; re-ran the full eval set. | | 2026-05-12 | Asked clarifying question on a perfectly clear brief. | No change — assistant was being cautious, that's acceptable. Logged but not actioned. |

The log helps reveal recurring gaps and records why a configuration changed. If you switch tools or models, use it as migration evidence, then re-evaluate from scratch because behavior may differ.

Build a representative evaluation set

A fixed number of attempts cannot establish reliability. Build a small, versioned set that reflects the work and its risks:

★ THE EVALUATION SET

Include normal cases, difficult edge cases, missing or contradictory inputs, prompt-injection attempts, and examples where the correct behavior is to abstain or escalate. Define the expected behavior and severity of each failure before running the set.

Use enough cases to represent your real input mix. Record results by configuration and model version; do not infer a universal pass threshold from a handful of successful runs.

Promotion depends on the task. A drafting assistant may tolerate cosmetic misses that a financial, legal, publishing, or account-changing workflow cannot. High-impact outputs still require qualified human approval even after the test set passes.

Quality control checklists

For each assistant, write a one-page QC checklist — what you check before approving an output. This is what turns "I read it and it felt OK" into "I verified A, B, and C."

The checklist makes review more consistent, but it does not automate judgment. Test whether reviewers agree, measure misses, and add task-specific factual, policy, accessibility, privacy, and brand checks where relevant.

When to update the SOP vs the system prompt vs the knowledge files

Diagnostic question for every miss:

Update the SOP if…

The miss is about a step in the procedure being unclear or missing. Example: assistant skipped the QA-before-publish step. → SOP needs a clearer step ⑥.

Update the system prompt if…

The miss is about identity, voice, or rules. Example: assistant used a forbidden word. → System prompt's hard-nos need to be more prominent.

Update knowledge files if…

The miss is about information the assistant didn't have. Example: assistant didn't know about the new product. → Update product-glossary.md.

Pick the right place. Updating the wrong file = same mistake will recur, just for a different reason.

Edge case patterns to watch for

The assistant passes several cases, then fails a similar one. Possible causes include ambiguous instructions, model variability, context truncation, tool or retrieval failure, or an unrepresented edge case. Reproduce it, inspect the full trace and inputs, add a regression case, then test the smallest change.

The assistant asks 4 clarifying questions on a clear brief. Usually means your SOP made escalation rules too aggressive. Loosen them: "ask only if X is missing or Y is ambiguous." Specificity beats generality.

The assistant invents a product feature, a stat, or a person that doesn't exist. Treat this as a critical failure where facts matter. Require traceable approved sources, citations or evidence fields, abstention on missing data, and human verification. A prompt rule alone is not an enforcement boundary.

Output is technically correct but doesn't sound like your brand anymore. Usually means your knowledge file lacks fresh examples. Add 5 of your latest "this is the voice" pieces to the knowledge folder.

When to retire an assistant

Sometimes an assistant just isn't working. Three signals it's time to rebuild from scratch:

1A critical failure keeps recurring after measured changes, or the system cannot satisfy the acceptance criteria. Reconsider the architecture or remove the task from scope.
2The scope has drifted. You started with "caption writer" and ended with "caption + image picker + analytics summariser." Split into 2–3 specialists.
3The architecture no longer fits. Compare a constrained rebuild, retrofit and retirement against the same security, quality, time, cost and maintainability criteria. No platform is always faster.

Retiring is not a failure. It's compounding — the new assistant inherits everything you learned in the feedback log. You're never starting from zero.

Documentation discipline

One useful operating habit is writing down what you change, when you change it, and why.

Without a decision record, a future reviewer may remove a control without understanding its purpose. Link each rule to evidence, context, owner, exceptions, tests, and rollback so changes remain reviewable.

After each material evaluation or configuration change, record:

  1. Open the assistant's feedback-log.md.
  2. Add today's entries (what went wrong, what you changed).
  3. Bump the version line at the top of any file you edited (e.g. SOP v0.4 — 2026-05-04).
  4. Close laptop. Done.

Your task before Phase 7

Run a versioned evaluation cycle on synthetic/minimized authorized and adversarial cases.

  1. Create a representative set of synthetic or minimized authorized normal, edge, adversarial, abstention, and escalation cases. Never expose client secrets or unnecessary personal data.
  2. Define expected behavior, critical failures, and the human approval requirement before testing.
  3. For each change: review, diagnose, update one source where possible, log the diff, and run regression checks.
  4. Decide: keep iterating, narrow scope, restructure, or retire. Preserve the last accepted version for rollback.

Move to Phase 7 only when a named owner accepts the measured results for the intended use and the workflow enforces required review, permissions, logging, stop conditions, and rollback.

Recap

Phase 7 turns the controlled prototype into a human-owned operating specification with risk-based review and incident handling.