When this framework fits
Use it when
- AI will classify, extract, retrieve, draft, rank, or recommend in a business workflow.
- Incorrect output has a measurable operational or customer consequence.
- Representative examples can be collected and judged.
Do not force AI when
- A deterministic rule solves the task more reliably.
- No one can define what a correct output looks like.
- The required data cannot be used safely or lawfully.
The pre-production evaluation framework
1. Define the task as inputs, outputs, and decisions
Replace "build an AI assistant" with a bounded job. Specify the input, required output schema, downstream user, and action the output may trigger. State what must never happen. A system that summarizes contracts needs a different standard from one that determines whether a payment is released. The evaluation must follow the consequence.
2. Establish a baseline worth beating
The baseline might be the current human process, a simple keyword search, a rules engine, a template, or a smaller model. Measure quality, time, and cost. Without a baseline, a sophisticated system can appear successful while adding expense and new failure modes to work that already performed well.
3. Build a representative held-out set
Collect examples across normal cases, rare but important cases, poor-quality inputs, and known failure conditions. Label them with clear guidance and measure reviewer agreement. Keep the final evaluation set separate from prompt and model development. If the team repeatedly looks at the test set and adjusts the system, it has become training data and no longer estimates generalization.
4. Choose metrics that match the error economics
Precision asks how many positive outputs were correct; recall asks how many true positives were found. Groundedness tests whether claims are supported by the supplied material. Exact-match or field-level accuracy can suit structured extraction. Ranking quality matters for search. Measure latency and cost per completed task, not only per model call. Weight errors by consequence rather than averaging them into one reassuring score.
5. Test the complete system
Evaluate retrieval, context assembly, model output, validation, permissions, and user interaction together. A strong model cannot recover documents that retrieval misses. A grounded answer can still expose information the user should not see. Structured outputs should be schema-validated. Citations should lead to the exact supporting passage, and missing evidence should produce abstention rather than invention.
6. Set release and deferral thresholds
Define the minimum performance for each critical class, not only the overall average. Route low-confidence, high-risk, or unsupported cases to a human reviewer. Specify what the reviewer sees and how correction feeds the evaluation set. The system should fail closed where the cost of a false positive is unacceptable.
7. Monitor quality drift after release
Track input mix, override rate, unsupported-answer rate, task completion, latency, and cost. Sample outputs for ongoing review and rerun the held-out set after changes to models, prompts, retrieval, or source material. Set thresholds that trigger investigation, rollback, or revalidation.
Extracting obligations from supplier contracts
A procurement team wants to identify renewal dates, notice windows, price-escalation clauses, and data obligations. A simple search baseline finds 72% of known clauses with 94% precision. The AI system must improve recall without sending invented obligations into the contract register.
The held-out set includes scanned files, amendments, conflicting terms, missing pages, and unusual drafting. Release requires at least 92% recall on renewal and notice clauses, at least 98% precision on dates, a citation for every field, and abstention when source text conflicts. Low-confidence cases go to legal operations. The system is not allowed to approve terms or provide legal advice. It creates a review queue with evidence.
Evidence required for a release decision
| Evidence | Decision it supports | Minimum discipline |
|---|---|---|
| Task definition and risk classes | What can be automated and what must defer | Named owner and prohibited outcomes |
| Baseline report | Whether AI creates incremental value | Same examples, metrics, and cost basis |
| Held-out evaluation set | Expected performance beyond the demo | Representative cases and label guidance |
| Error analysis | Whether failures are acceptable and fixable | Segment by consequence and root cause |
| Cost and latency tests | Whether the workflow works at real volume | Measure end-to-end completed tasks |
| Monitoring and rollback plan | Whether quality can be operated over time | Thresholds, owners, and response playbooks |
Common failure modes
Demo-set overfitting: polished examples are mistaken for representative performance. No baseline: model quality is celebrated without showing operational improvement. Average-score hiding: a strong overall result conceals failure on the highest-risk class. Retrieval blindness: answer quality is measured only when the right source is already present. Unowned deferral: the model routes uncertainty to a queue nobody staffs. Cost omission: retries, review, retrieval, and monitoring are excluded from cost per task. No drift plan: quality is assumed to remain fixed as inputs change.
Pre-production checklist
- The task, output schema, user, decision, and prohibited outcomes are defined.
- A simple rule, search, or current process is measured as the baseline.
- The held-out set covers ordinary, difficult, and high-consequence cases.
- Metrics reflect the costs of false positives, false negatives, and unsupported claims.
- Retrieval, permissions, validation, and user workflow are tested end to end.
- Deferral thresholds, review ownership, and escalation timing are explicit.
- Cost, latency, monitoring, rollback, and reevaluation are part of release.
Limitations
An evaluation estimates performance on the represented distribution, not every future input. Labels can encode reviewer disagreement, and rare risks may not appear often enough to measure statistically. Model and vendor behavior can change. High-consequence uses may require domain validation, legal review, security assessment, and stronger controls than this general framework describes. The correct release may be a narrow assisted workflow, not autonomous action.
Related services and cases
Set the release standard before the demo
We can design the evaluation, guardrails, and operating thresholds around your real workflow.