OpenAI released the GPT-5.6 family on July 9, and Google DeepMind updated its Gemini 3.6 Flash model card on July 21. Both launches emphasize capability and efficiency. Neither vendor can tell you which model will correctly explain your refund exception, stay inside your support policy, and respond fast enough on a mobile checkout page.
Use this five-step chatbot model evaluation before changing production: freeze the prompt and sources, build a 20-case test set, run every case three times per model, grade outputs without model names, and reject any candidate that fails a hard safety gate before comparing averages. The scorecard below turns that sequence into a decision your team can repeat when the next model arrives.
Write the Decision Before Running the Models
Start with one sentence: “We are deciding whether model B should replace model A for this chatbot, for these visitors, under these constraints.” Name the chatbot and traffic segment. A model that is excellent for internal technical support may be a poor fit for a public sales assistant.
Record the production baseline before testing:
- current model, prompt version, and source snapshot;
- median and 95th-percentile response time;
- cost per 100 completed conversations;
- answer, handoff, abandonment, and negative-feedback rates;
- known high-risk intents and required refusal behavior.
Then define the minimum improvement worth a switch. “The new model sounds nicer” is too vague. A useful threshold might be: maintain every billing-policy gate, reduce unsupported claims by at least 30%, keep median response time below three seconds, and add no more than $4 per 100 conversations.
This decision frame prevents benchmark drift. Without it, reviewers tend to reward whichever answer is longer, more confident, or written in the style they personally prefer.
Build a Test Set That Resembles Your Traffic
Twenty cases are enough for a first comparison if each one has a purpose. Use real, redacted customer questions rather than generic trivia. Include the easy traffic that drives volume and the rare cases where a wrong answer creates a refund, privacy, or account-access problem.
| Field | What to record | Example |
|---|---|---|
case_id | Stable identifier | billing-07 |
user_message | Redacted, production-shaped wording | “Can you move last month's invoice to my company?” |
intent | One operational label | invoice-correction |
expected_facts | Claims the answer must include | Issued invoices cannot be edited in the dashboard |
forbidden_claims | Claims that fail the case | Finance will definitely reissue it |
required_action | Answer, clarify, refuse, or hand off | Hand off to billing review |
expected_source | Governing document or Q&A ID | billing-policy-v4 |
risk_weight | Consequence multiplier from 1 to 5 | 4 |
A balanced 20-case set can contain eight common questions, four paraphrases of common questions, four missing-or-conflicting-source cases, and four high-risk boundaries. Do not let every case be a direct FAQ lookup. Customers omit dates, mix two intents, quote an old policy, misspell product names, and ask the bot to make exceptions.
Keep a protected slice that the person tuning prompts does not see. The hidden evaluation set guide explains how to keep a test from becoming another prompt template. Even a small sealed set helps reveal whether the candidate improved broadly or merely learned the visible examples.
Hold Everything Except the Model Constant
For the first round, use the same system instructions, sources, retrieval settings, tools, conversation history, temperature, and output limits. Change only the model. Otherwise, you are comparing bundles and cannot explain the result.
Capture the exact context supplied on each run. If one candidate receives a newer policy chunk, its apparent reasoning advantage may be a retrieval difference. If a provider applies a different default output limit, a truncated answer may look like a quality failure. Normalize the settings you control and log the differences you cannot.
Run each case at least three times. Language-model output varies, and one polished response can hide an unstable pattern. Three runs will not establish a perfect statistical estimate, but they will expose candidates that alternate between the correct policy and a confident invention.
Use fresh conversations for independent cases. Then add a smaller multi-turn set for follow-up behavior: corrections, pronouns, changed requirements, and a user who refuses the proposed next step. Mixing these into one score obscures whether a model fails on source grounding or conversation state.
Grade With Hard Gates and a Short Rubric
Binary checks should decide whether an answer is eligible. Averages should decide among eligible answers.
Set hard failures for behaviors that cannot be offset by great tone elsewhere:
- inventing a price, deadline, entitlement, or completed action;
- using the wrong customer's data or exposing protected information;
- ignoring a required refusal, identity check, approval, or handoff;
- citing a source that contradicts the answer;
- calling a destructive or chargeable action without confirmation.
After those gates, score the dimensions that matter to the customer. Keep the rubric small enough that two reviewers can apply it consistently.
| Dimension | 0 | 1 | 2 | Weight |
|---|---|---|---|---|
| Correctness | Wrong or contradictory | Partly correct | Fully correct | 5 |
| Grounding | Unsupported | Source is relevant but incomplete | Claims follow from the governing source | 5 |
| Boundary control | Breaks policy | Safe but vague | Correctly answers, clarifies, or hands off | 5 |
| Task completion | No usable next step | Partial next step | Clear, executable next step | 3 |
| Clarity | Confusing or bloated | Understandable | Direct and easy to scan | 2 |
The maximum weighted score is 40. Calculate weighted points / 40 × 100 for each run, then multiply the case result by its risk weight when computing the overall quality score. Also report the unweighted score. If the two disagree sharply, a candidate may be improving low-risk prose while regressing on the cases that matter most.
Blind reviewers to model names and randomize left-right order. Reviewers bring vendor expectations into close calls, especially when one model tends to write longer answers. For subjective dimensions, have two people independently grade the first five cases, compare disagreements, and tighten the rubric before grading the rest.
Worked Evaluation: The Cheaper Model Loses on One Rule
Assume an online retailer compares its current model, A, with a faster candidate, B. Both receive the same returns policy:
Standard items may be returned within 30 days. Damaged items may be reported within 60 days for review. Support must approve requests submitted after 30 days.
One risk-weight-5 test asks:
Customer: My order arrived damaged 45 days ago. Can you refund it today?
Model A answers:
Your damaged order is still within the 60-day reporting window. I cannot promise an immediate refund because support must review requests made after 30 days. I can help you send the order details to support.
Model B answers:
Yes. Damaged orders can be refunded within 60 days. Send your order number and I will start the refund.
Model B is shorter and sounds decisive, but it promises both approval and an action the source does not authorize. That is a hard failure, not a one-point deduction for wording.
After 60 runs per model—20 cases, three runs each—the summary looks like this:
| Result | Model A | Model B |
|---|---|---|
| Hard-failure runs | 0 | 3 |
| Unweighted quality | 84/100 | 88/100 |
| Risk-weighted quality | 87/100 | 81/100 |
| Median response time | 2.8 s | 1.7 s |
| Estimated cost per 100 conversations | $6.40 | $3.90 |
Model B wins on speed, cost, and the unweighted average. It still does not qualify as the default because all three hard failures involve unauthorized billing or refund commitments. The team can test B on a low-risk FAQ route, revise the prompt and action boundary, or reject it. It should not average its way into handling refunds.
This is why a single “win rate” is not enough. The score must preserve the reason a response lost.
Measure Latency and Cost on Complete Conversations
Provider prices and public benchmarks are useful for selecting candidates, not for making the production decision. Your prompt length, retrieved context, answer length, cache behavior, retries, and tool calls determine the bill customers actually create.
Measure at least:
- time to first visible token;
- time to a complete answer;
- input and output tokens per turn;
- retries, tool calls, and failed tool calls;
- cost per completed conversation, not only per message;
- abandonment and handoff after slow responses.
A cheaper model that needs two clarification turns can cost more per resolution than an expensive model that answers once. A fast first token followed by a 20-second tool call may still feel broken. The chatbot speed guide gives latency the full treatment; in a model evaluation, keep speed beside correctness so neither disappears into one blended number.
Do not invent a universal weighting formula for cost, speed, and quality. Apply hard gates first, then compare the eligible candidates against the threshold written at the start. A high-volume FAQ bot may reasonably choose the cheapest model above its quality floor. A finance or account-security bot should accept more latency and cost for materially better boundary control.
Use Side-by-Side Testing Without Fooling Yourself
Side-by-side output makes differences easy to inspect, but it creates its own bias. The longer answer often looks more thorough. The first answer anchors the reviewer. A recognizable writing style can reveal the vendor even when the model name is hidden.
Use the comparison view for diagnosis, then score each response against the saved rubric. Do not ask reviewers which answer they “like.” Ask whether the expected facts appear, forbidden claims are absent, the required action occurs, and the wording earns its clarity points.
In Agentkit, the Playground's Compare tab sends the same question to two or three available models side by side. Keep the instructions synchronized for the controlled round. After identifying finalists, use Show Sources in the Playground to verify which documents support individual answers. The comparison surface speeds up inspection; your saved cases and rubric make the result reproducible.
The broad GPT vs. Claude vs. Gemini guide can help form a shortlist. Your own cases should choose the winner because support policies, visitor language, and failure costs are specific to your deployment.
Make the Switch Reversible
Save the evaluation date, candidate model IDs, settings, source snapshot, raw outputs, grades, reviewer notes, latency, token use, and decision. Model behavior and provider infrastructure change, so “we tested it last quarter” is not enough evidence unless the run can be reconstructed.
After the offline evaluation, expose the winner gradually if your deployment supports it. Watch the exact intents that distinguished the candidates, along with handoffs, negative feedback, repeated questions, latency, and cost. Keep the old configuration ready until the production sample clears the same gates.
If the candidate regresses, restore the known configuration first and investigate second. The chatbot model rollback checklist covers ownership, triggers, and rollback evidence for that moment.
Choose the Model Your Customers Can Safely Use
Model launches arrive with impressive aggregate results. Your chatbot needs a narrower proof: it answers your common questions, respects your policies, handles missing evidence, and stays within your latency and cost constraints across repeated runs.
Write the decision before testing. Hold the system constant. Let hard failures eliminate unsafe candidates, then use risk-weighted quality, complete-conversation latency, and real cost to choose among the survivors. Save the artifacts so the next comparison starts with evidence instead of memory.
In Agentkit, side-by-side model comparison, source inspection, and conversation logs support the evaluation loop from controlled tests through production review.
No credit card required.
