OpenAI launched Presence in July 2026 with a striking result: its support deployment reportedly met or exceeded frontline human-support benchmarks and resolved 75% of inbound issues without human help. That is evidence about one managed deployment, not a portable target for every support team.
Before running your own AI support benchmark, write down seven fields: the decision, eligible cases, assignment method, shared evidence, durable outcome, hard safety gates, and launch rule. If any field is missing, the final percentage will be easy to celebrate and hard to trust. The benchmark contract below is the practical asset to complete before anyone sees a chatbot score.
Write the Decision as a Benchmark Contract
Start with the operational decision, not a broad question such as “Is AI better than people?” A useful benchmark decides whether one defined workflow should move to a chatbot-first path for one defined audience.
Here is a contract for a subscription company evaluating billing support:
decision: chatbot-first routing for routine billing questions eligible_cases: - invoice copy requests - payment status questions - disputed charge questions excluded_cases: - suspected account takeover - active legal or regulatory complaints assignment: random within intent, language, and customer tier shared_evidence: - billing policy version 14 - current invoice and payment status - identity verification result durable_resolution: no repeat contact for the same issue within 72 hours hard_gates: cross_account_disclosure: 0 unauthorized_financial_commitment: 0 policy_defect_rate: "< 1%" launch_rule: automate only intents that pass every gate and reach human-baseline durable resolution within 3 percentage points
Keep this contract fixed while the benchmark runs. If the team changes the included intents, source material, reopen window, or passing threshold after seeing results, it has created a new experiment. Save that as a new version instead of silently editing the original.
The contract also prevents a common category error. A model evaluation asks which model performs better under controlled settings. The chatbot model evaluation guide covers that comparison. An operational benchmark asks whether the entire chatbot path—including retrieval, identity, routing, handoff, and recovery—produces outcomes comparable to the existing support path.
Compare Matched Work, Not Two Average Queues
The chatbot and human cohorts need the same work mix. Sending password resets and invoice copies to the chatbot while humans handle fraud, angry customers, and undocumented exceptions guarantees a flattering but meaningless comparison.
Create strata before assignment. At minimum, separate cases by:
- intent: the customer job, not a broad department label;
- complexity: routine lookup, documented exception, or judgment-heavy case;
- risk: informational, account-specific, financial, privacy, or access-changing;
- language and channel: because source coverage and turn-taking differ;
- customer tier or product: when policies, entitlements, or service levels differ;
- time window: so an outage, campaign, or policy launch affects both paths.
Then assign cases within each stratum. Random assignment is strongest when operations allow it. If randomization is impractical, match cases from the same period using the same intent, risk, language, and customer tier. Record the matching rule before selection and keep unmatched cases visible; removing difficult cases after review makes the benchmark look cleaner than the service actually is.
Use enough cases to expose material failures. A 50-case run can reveal broken sources and obvious boundary errors, but it cannot establish that a 1% defect rate is stable. Start small for debugging, then size the decision run around the rarest failure that would stop launch. If unauthorized commitments are the main risk, the benchmark needs enough relevant financial cases to test that boundary directly.
Give Both Paths the Same Truth
A fair comparison does not require identical interfaces. It requires equivalent authority and evidence.
Both paths should receive the same policy version, current account data, identity status, and allowed actions. A human may use a support console while the chatbot calls an API, but neither should see a newer refund rule or a more complete customer record. Log the source version and retrieval time with every case so a later review can distinguish a reasoning failure from an evidence mismatch.
Do not give the chatbot hidden hints that humans never receive, such as the correct intent label or expected answer. Do not handicap it by withholding structured data that the production integration would provide. The benchmark should represent the service you can actually operate.
Use one answer key for grading both paths. The ground-truth workflow shows how to record governing evidence, allowed outcomes, forbidden claims, and expiry rules. The grader should be able to mark the same unsupported promise as a defect whether a chatbot or a person wrote it.
Blind review where practical. Remove the responder identity, normalize obvious formatting differences, and ask reviewers to grade facts, boundary decisions, task completion, and next steps. Perfect blinding may be impossible because chatbot prose has recognizable patterns, but a fixed rubric still reduces preference-driven scoring.
Measure Durable Outcomes and Hard Failures Separately
One average cannot carry quality, safety, customer effort, and cost. Keep hard failures separate, then compare the surviving paths on durable outcomes.
| Measure | Formula | Why it belongs in the benchmark |
|---|---|---|
| Durable resolution rate | Cases resolved with no same-issue contact inside the reopen window / eligible cases | Stops a quiet exit from counting as success |
| Policy defect rate | Responses that contradict governing policy / reviewed responses | Reveals confidently wrong service |
| Correct handoff rate | Required handoffs sent to the right queue with complete context / required handoffs | Credits safe routing instead of punishing every escalation |
| Median customer turns | Customer messages before resolution or accepted handoff | Detects hidden effort behind a “resolved” label |
| Cost per durable resolution | Total cohort cost / durable resolutions | Includes retries, review, and human fallback |
Choose a reopen window that matches the workflow. Seventy-two hours may suit billing questions; a delivery promise may need seven days; an annual renewal issue may need longer. Apply the same window to both cohorts.
Hard gates do not average away. Cross-account disclosure, an unauthorized refund promise, or a completed action without confirmation should fail the relevant intent even if hundreds of routine answers were correct. Record the case, governing rule, responder path, evidence supplied, and customer impact. A zero-tolerance gate requires exact incident review, not a rounded dashboard percentage.
Customer satisfaction is useful but incomplete. People may reward speed while missing a policy error, or rate a correct refusal poorly because they dislike the outcome. Keep CSAT beside the evidence-based grade and durable resolution; do not use it as a replacement for either.
Worked Benchmark: The Overall Average Hides the Launch
Suppose a subscription business assigns 1,000 billing conversations evenly between chatbot-first and human-first support. Assignment is randomized within three intents, and both paths use the same policy, identity result, and billing records.
After the 72-hour reopen window, the durable results are:
| Intent | Cases per path | Chatbot durable resolution | Human durable resolution | Chatbot defects | Decision |
|---|---|---|---|---|---|
| Invoice copy | 250 | 225 (90%) | 220 (88%) | 0 | Launch chatbot-first |
| Payment status | 150 | 112 (74.7%) | 116 (77.3%) | 0 | Retest after source fix |
| Disputed charge | 100 | 42 (42%) | 68 (68%) | 3 | Keep human-first |
| Total | 500 | 379 (75.8%) | 404 (80.8%) | 3 | Do not launch globally |
The global chatbot result looks respectable, and invoice copies are already above the human baseline. A single rollout decision would still be wrong. The three disputed-charge defects include two unsupported refund promises and one claim that a pending charge had been reversed. That intent fails its hard gate regardless of the overall score.
Payment status falls only 2.6 percentage points behind the human path, inside the contract's quality margin, but review finds that most misses involve bank-transfer timing absent from the chatbot's source. The team adds the approved timing policy, creates regression cases, and reruns that stratum instead of changing the prompt blindly.
Now add cost. The full chatbot cohort costs $730, including model usage, review, and human fallbacks; the human cohort costs $2,600. Cost per durable resolution is about $1.93 for the chatbot path and $6.44 for the human path. That advantage supports launching invoice copies, but it cannot purchase permission to automate disputed charges. The benchmark produces an intent-level routing policy: automate the proven slice, repair and retest the near miss, and retain people where consequence and quality still diverge.
Turn the Result Into a Routing Rule
End the benchmark with a decision for every stratum: launch, repair and retest, keep human-first, or stop because the evidence is insufficient. Assign an owner and an expiry date. Policies change, customer language shifts, and a passing route can decay after launch.
For launched intents, keep a small random review sample plus targeted queues for reopens, negative feedback, repeated questions, and handoffs. The chatbot QA sampling plan explains how to preserve a representative baseline without letting rare risks disappear. Compare the same durable metrics weekly during the first release window, then monthly once performance stabilizes.
Do not silently expand the route from “invoice copy” to “billing.” New intents need their own evidence. If the chatbot starts receiving charge disputes because a keyword rule is too broad, the production population no longer matches the benchmark contract.
Re-run the affected strata after a material model, prompt, source, action, identity, or routing change. Preserve the prior contract and raw results so the team can tell whether the new release improved service or merely changed the measurement.
Make the Benchmark Answer a Real Deployment Question
The useful conclusion is rarely that chatbots or humans are universally better. It is a precise boundary: which customer jobs the chatbot can handle at comparable quality, which ones need repair, and which ones still require human judgment or authority.
Agentkit conversation logs and analytics can supply review evidence, while Q&A pairs can pin corrected answers after an approved source fix. Keep the benchmark contract outside the model, grade both paths against the same truth, and launch only the intents whose durable results earn it.
No credit card required.
