OpenAI introduced GPT-6.1 Sol on September 29, citing stronger performance and lower cost on several demanding tasks. The launch is a reason to retest a support chatbot, not evidence that a new model will answer your French shipping policy or Japanese cancellation question correctly. Vendor benchmarks rarely contain your sources, your customer phrasing, and your escalation rules.
Before switching models, make a language-by-intent test sheet. For each important language, include a routine answer, a regional policy exception, a missing-source question, and a request that should reach a human. Record the source and expected action before running either model. The release gate is simple: no new critical failure in any language, no drop below the agreed quality floor in any language, and a measured cost or latency gain on the traffic you actually serve.
Choose the languages that carry real risk
Start with conversation volume, but do not let volume make the whole decision. A language used by 3% of visitors can still carry a policy that differs sharply from your English default. Pick your top languages, then add any locale with different prices, returns, eligibility, identity checks, or handoff coverage. Keep country and language as separate fields. French spoken in Canada does not automatically imply the same policy as French spoken in France.
Pull recent, consent-appropriate questions from chat logs or support tickets. Remove names, addresses, order numbers, and other personal data. Preserve the customer's wording where possible. A polished translation of an English test prompt often misses the shorthand, spelling, and local terms that caused the original failure. Have a reviewer who knows the language rewrite or approve each case.
The first test set can be small. Say you serve English, French, German, and Japanese. Six support intents per language, with a routine and an awkward variant for each, produce 48 cases. Add one missing-source case and one escalation case per language for 56. That is enough to catch obvious regressions before a limited rollout. Grow the set with real misses instead of multiplying generic paraphrases.
The multilingual support guide explains why translated answers still need local sources and handoffs. Here, those requirements become test cases for a model change.
Write the expected result before reading model output
A model can write natural French and still apply the wrong return window. Grade the business decision separately from the language. Each test case needs a source snapshot, a locale, and an expected outcome that a reviewer can check without judging style.
Use these fields as a compact answer key:
| Field | What to record | Example |
|---|---|---|
| Locale and market | Language plus applicable country or region | fr-FR, France |
| Customer intent | The decision the visitor wants | Return an opened accessory |
| Source version | Exact policy or document revision | returns-fr-2026-09-18 |
| Required facts | Facts the answer must include | Opened accessories need human review |
| Forbidden claim | Statement that makes the answer unsafe | "You are automatically eligible" |
| Expected action | Answer, ask, or hand off | Ask for order reference; route to support |
Do not let a model generate its own answer key. The source owner and a language reviewer should approve it. If the policy is unclear, fix the source or mark the case unresolved. A disagreement between models cannot settle an ambiguous company rule.
Keep the source version with the case. When a policy changes, update both the answer key and the chatbot's knowledge. Otherwise you may reject a correct new-model answer because the test still expects last month's rule. The ground truth guide goes deeper on maintaining that answer key.
A worked French returns case
Consider a fictional retailer with this approved French policy: unopened accessories may be returned within 30 days; opened accessories require a support review. The English policy gives 45 days for unopened accessories. The difference is deliberate. The French visitor asks:
J'ai ouvert l'accessoire hier. Puis-je le renvoyer automatiquement ?
The test record sets locale=fr-FR, market=FR, source=returns-fr-2026-09-18, and expected_action=handoff. A passing answer in French says that opened accessories require review, asks for the order reference through the approved support path, and avoids promising approval. It need not quote the policy word for word.
Now compare two hypothetical outputs:
Current model: "Les accessoires ouverts doivent être examinés par notre équipe. Envoyez votre référence de commande au support pour vérifier les options."
Candidate model: "Oui, vous pouvez le retourner automatiquement sous 45 jours."
The candidate sounds fluent. It also imports the English window and invents automatic eligibility. Mark the case as a critical failure even if a bilingual style grader prefers its wording. Review the retrieved source alongside the response. If the chatbot supplied the French policy and the model ignored it, the model or prompt failed. If retrieval supplied the English page, repair retrieval before making a model decision. That distinction prevents a model comparison from hiding a source-routing bug.
Run the same case with a visitor who asks in English while shopping in France. The applicable market should still be France. Language detection and policy selection are different decisions; testing only French-language prompts misses that boundary.
Score decisions before polish
Use a short rubric that a reviewer can apply consistently. Start with factual and operational checks. Then assess whether the answer is clear and natural. The latter matters, but it cannot erase a wrong policy or a missed escalation.
For each case, record four results: source correctness, decision correctness, action correctness, and language quality. Score the first three as pass or fail against the answer key. Score language quality on a small scale such as 0 to 2: unusable, understandable but awkward, or clear and locally appropriate. Keep the raw response and reviewer note so a second reviewer can challenge a borderline judgment.
Declare hard failures in advance. Examples include applying the wrong market's policy, exposing another customer's data, fabricating an order status, promising an unauthorized refund, or failing to hand off when the source demands review. A single new hard failure blocks the switch for that locale. This is stricter than an average score, and it should be. An average can improve while a rare but expensive case gets worse.
For routine cases, compare rates by language and intent. An overall pass rate weighted by volume is useful for forecasting impact, but it must sit beside the worst-performing language. A model that gains five points in English and loses ten in German is not an uncomplicated upgrade for a German storefront.
OpenAI's September customer-support optimization example recommends testing the same representative cases for quality, latency, and cost before accepting savings. It explicitly calls for language and region coverage in production samples. Use that principle with your own support policies instead of copying a vendor's sample tickets.
Keep the comparison fair
Run the current and candidate models against the same frozen case set. Match the system instructions, retrieved documents, tools, output limits, and available customer state as closely as the providers permit. Log every difference you cannot eliminate. If the candidate gets a newer source or a shorter tool payload, you are comparing two application configurations, not only two models. That may still be a useful test, but label it honestly.
Repeat cases that produce variable answers. A single pass can miss an intermittent wrong-policy response. For high-risk cases, run each prompt several times and inspect the worst answer, not only the most common one. Keep any automated grading narrow: a deterministic check can spot a forbidden 45-day claim, while a human reviewer decides whether the full response gives the right next step. A second model acting as a grader can help sort a queue, but it should not be the sole judge of locale-specific policy.
Measure response time at the point the visitor experiences it. Record time to first useful answer and time to complete answer, with p50 and p95 by locale. Translation, retrieval, and routing may dominate the delay. Record model tokens and billed cost per completed conversation as well as per request. A cheaper single turn can require an extra clarification turn and cost more overall.
The model evaluation guide covers a broader cross-model scorecard. For multilingual support, the key addition is a separate acceptance decision for each locale and market.
Read a result without hiding the weak locale
Suppose the current model passes 51 of 56 cases, while the candidate passes 52. That one-case overall gain looks good on a slide. The per-language results tell a different story:
Current Candidate English 13/14 14/14 French 13/14 12/14 German 12/14 13/14 Japanese 13/14 13/14 Total 51/56 52/56
If the candidate's new French miss is the opened-accessory case above, the switch fails the hard gate for France. Lower token cost does not pay for a wrong eligibility promise. Keep the current model there while you inspect retrieval, prompts, and the candidate's response. You may still run a controlled trial in another market if its own gate passed and your routing can keep the models separate.
Also look at confidence in the comparison. Fourteen cases per language are a smoke test, not a precise estimate of production accuracy. Add real conversations from the failure cluster and rerun the frozen set. Avoid claiming a one-point gain is statistically meaningful. The purpose of the small set is to stop an obvious regression early and tell you where deeper sampling is worth the effort.
Release by locale, then keep watching
After the offline gate, route a small share of eligible traffic to the candidate in one approved locale. Preserve a way back to the current model. Monitor the exact failure types you tested: wrong-market facts, unsupported promises, needless refusals, handoff rate, repeat contact, and latency. Review actual conversations daily at first, especially those involving regional policies. A customer complaint should become a new test case with a source version and owner.
Do not treat an increase in escalation as automatically bad. The current model might have answered questions it should have handed off. Compare the reason for each escalation and whether the human received enough context to continue. Likewise, a lower escalation rate may reflect unsafe confidence rather than better resolution.
If you cannot route by market, keep the old model until every served locale passes. If you can route, document which model handles each market and what happens when language or market is unknown. A mixed-language conversation should not silently cross to a different policy. Test that transition with the same care as the first answer.
Model launches will keep arriving. A small, reviewed set of local customer questions makes each switch a measured release decision instead of a bet on a benchmark. Keep the cases, sources, and failure notes; they become more valuable with every model comparison.
Agentkit supports automatic language detection across 95+ languages, offers a model picker, and keeps conversation logs that can inform the next review. Use your own policy owners and language reviewers to approve the answer key before changing a live chatbot.
No credit card required.
