Use a paired review to decide whether a model upgrade can ship when overall quality improves but a few important workflows get worse.
The candidate model wins on 64 of 100 prompts. A reviewer notices that it also invents an order status in two examples where the original model correctly asked for more information. The average score improved. The team now needs to decide whether these two changes are acceptable.
One way to make that decision is to review each disagreement as a pair. Hold the input and evidence constant, compare the two outputs against the expectations for that task, and distinguish a release-stopping failure from a preference. This is especially useful for a model upgrade within a product that customers already use. They will notice the behaviors that changed even if a general leaderboard improves.
Compare the outputs people will actually see
Collect examples from a defined workflow, with permission to use the data and with sensitive details removed where appropriate. Fix the retriever results and tool responses for the first pass so the reviewers can attribute a change to the model or its prompt. Run a separate end-to-end pass afterward. If the new model calls tools differently, that difference belongs in the second pass. Save the configuration of both versions and carry it into the review so someone can reproduce a surprising output.
Use short case instructions. For instance, an order assistant must name a delivery date only when the supplied tracking record contains one. Otherwise it may state that the date is unavailable and suggest a next check. A reviewer can apply that instruction without inferring the team’s intent. The same prompt may also benefit from a concise tone, but tone belongs in a different rubric row from an unsupported operational claim.
Make disagreements easy to adjudicate
OpenAI’s evaluation best practices discuss pairwise comparisons and point out position and verbosity biases in model judging. In a human review, shuffle which output appears first and hide version labels for the initial decision. Give both answers the same visual treatment. If you use a model to triage the queue, validate its judgments against people on a sample. Have those people review cases in which the grade would stop a release.
| Disagreement in a paired review | Evidence needed | Suggested release decision |
|---|---|---|
| Candidate adds an unsupported customer commitment | Original facts and exact commitment text | Block the affected workflow until corrected |
| Candidate asks a clarification where the original guessed | The missing field and policy for asking | Credit the improvement if the clarification is specific |
| Candidate uses more words with the same facts and action | Task length target and user context | Record a style preference or measurable verbosity issue |
| Candidate gets the right answer after an unauthorized tool choice | Trace of the actual tool call and its scope | Block the tool path and retest separately |
| Reviewers disagree about what the task requires | Case rule and each reviewer’s rationale | Resolve the rubric gap before counting the case |
Give reviewers a way to select equivalent and insufficient evidence. A forced preference invites someone to turn a minor phrasing change into a quality result. When outputs truly differ, save the reason rather than just the winner. In a later release the team will need to know whether a fix changed the facts, the tone or the action.
Interpret failures by exposure and consequence
In the example, suppose both unsupported dates occur when the lookup returns a delayed shipment with an empty date field. Verify a few new examples of that pattern before changing the prompt. Then test the original model on those same cases so the team knows whether the issue is a regression or an existing weakness. Count the failure separately from the headline win rate. Ask the service owner what would happen if a customer relied on the date and whether the affected route can remain on the original model.
A product manager might accept changes to wording with a review after launch. A case that promises a nonexistent refund can require a hard release gate. Define those gates before the final tally. If the team rewrites a gate after seeing the outcome, record the reason and run the reviewers through the updated definition. Otherwise the release decision can drift with whoever is in the room.
Keep a compact release record
Record the input set, fixed evidence, candidate and baseline versions, paired results and unresolved findings. Add the name of the person responsible for each follow-up. For each critical regression that was fixed, keep a repeatable test and an example close enough to catch the pattern again. A model that eventually answers one exact prompt correctly can still fail on a differently worded request.
Roll out first to a workflow whose cases reviewers can inspect within the pilot period. Agree on what would pause routing and how people reach the prior configuration. Compare a sample of actual disagreements with the offline examples. If all field failures involve missing timestamps and the pilot review has mainly assessed tone, put those incidents into the next test set. Give the on-call owner the record so the decision can be reconstructed during an incident without retelling the whole evaluation.
Include at least a few cases where the baseline also fails. Reviewers can then tell whether they are choosing the better of two inadequate outputs or seeing a real improvement in readiness. In one example, both models may omit a required audit reference, while one gives a more pleasant answer. Give that pair a failure in the required field and a separate style observation. During sign-off, review the number of unresolved baseline issues that the rollout would inherit. Decide who owns them regardless of which version ships, so a successful comparison does not remove them from the backlog.
Emerging Stacks can build paired test sets and review workflows through AI and machine learning services. Contact Emerging Stacks with an upgrade example where reviewers disagree about which output should ship.
Ready to work with us?
Get in Touch


