OpenAI's July launch of Presence put production sessions, escalations, and quality signals at the center of improving customer agents after launch. Recent evaluation research points in the same direction: PACE found that a carefully selected proxy set could predict expensive agent-benchmark results at less than 1% of the full evaluation cost. Neither result means a small sample is automatically trustworthy. Selection is the work.
Use two review queues every week. Draw a random baseline from all eligible conversations to estimate the direction of overall quality. Build a separate targeted risk sample to investigate rare, expensive, or newly changed behavior. Report them separately.
| Queue | Include | Use the result for | Never use it for |
|---|---|---|---|
| Random baseline | Every eligible conversation has a known chance of selection | Directional defect, resolution, and escalation trends | Diagnosing enough examples of a rare failure |
| Targeted risk | High-risk intents, actions, escalations, negative feedback, long loops, new releases | Finding failure modes and creating fixes | Claiming an overall defect rate |
That two-queue design is the sampling plan to keep. The rest of this guide shows how to define the population, choose cases, review them consistently, and turn an 80-conversation week into evidence rather than theater.
Start With the Decision the Sample Must Support
A review sample can answer several questions, but one number cannot answer all of them.
An operations lead may need to know whether answer quality is improving across the whole chatbot. A billing owner may need to find every way the bot mishandles refund exceptions. A release owner may need early evidence that a new shipping source caused a regression. A support manager may need to understand why qualified handoffs arrive without useful context.
Write the decision before selecting conversations:
Decision: Keep the new refund-policy source at 100% traffic or roll it back. Population: English website conversations that reached a refund intent. Window: First 72 hours after release 2026-08-09.1. Primary failure: Answer contradicts the current refund policy. Stop condition: One unauthorized refund promise or a targeted pass rate below 95%. Owner: Support operations.
This card makes the sample auditable. It prevents a reviewer from quietly changing the question after seeing the results. It also separates a release gate from the ongoing random baseline; the release sample is intentionally concentrated on the changed path.
The broader AI chatbot QA workflow explains how to label answers and turn defects into tests. Sampling sits one step earlier: deciding which production conversations deserve that review and what conclusions the reviewed set can support.
Define the Population Before Applying Filters
“Review 50 chats” is incomplete until the team knows which chats could have been selected. Start with an export or query that fixes the population and preserves a count before any quality filter is applied.
Record the chatbot version, channel, time window, languages, visitor states, and excluded traffic. Remove internal testing, known synthetic monitors, duplicate retries, and empty sessions using stable rules. Do not remove short, abandoned, or apparently successful conversations merely because they look uninteresting. Those cases may be where the chatbot created friction without attracting a thumbs-down.
Give each eligible conversation a stable ID. For the baseline, select IDs randomly with a recorded seed or another reproducible method. “Scroll until something looks wrong” is defect hunting, not random sampling.
Watch for unit mistakes. A conversation with 18 messages should not receive 18 chances to enter a conversation-level sample. If the question concerns individual answers, define the unit as an answer and keep the parent conversation attached. If it concerns customer outcomes, sample whole conversations so the reviewer sees repeated questions, handoffs, and whether the user returned to the same issue.
Sampling frames also expire. A Friday export cannot represent a source update released Saturday. Save the query, selection time, population size, and version boundary beside the review results.
Keep a Random Baseline and Targeted Queues
The baseline protects against selection bias. Randomly chosen conversations include quiet successes, silent abandonments, routine questions, and failures with no explicit feedback. Even a modest sample can reveal whether the quality trend moved enough to investigate, as long as the team labels it as directional rather than precise.
Targeted queues trade representativeness for discovery. Useful queues include:
- pricing, refunds, privacy, security, and other high-consequence intents;
- conversations that attempted an API action, form, booking, or account change;
- human handoffs, especially repeated transfers or missing routing fields;
- negative ratings, repeated questions, fallback loops, and abrupt abandonment;
- conversations on a newly changed prompt, source, model, action, or channel;
- low-volume languages, customer segments, and edge cases that random selection rarely reaches.
Make the queues mutually exclusive when you need clean counts. Assign each conversation to the first matching category in a fixed priority order, such as high-risk action, policy intent, escalation, negative signal, new-release path, then remainder. If the same case can appear in several investigative queues, deduplicate it before calculating reviewer workload and retain all applicable tags.
Do not merge targeted pass rates into the baseline. If half the review set was selected because customers clicked thumbs-down, the combined defect rate is designed to look bad. The sample can still reveal valuable defects; it simply cannot estimate their prevalence.
Worked Sample: 80 Reviews From 12,000 Conversations
Suppose a support chatbot handled 12,000 eligible conversations last week. The team can review 80. It allocates 30 to the random baseline and 50 to targeted investigation.
| Review slice | Population signal | Reviews | Defects found | Valid conclusion |
|---|---|---|---|---|
| Random baseline | All 12,000 eligible conversations | 30 | 3 | Overall quality may be near 90%; accumulate more weeks before claiming precision |
| Billing and refund risk | 240 conversations | 12 | 4 | Four concrete policy failures need classification and fixes |
| Human escalations | 960 conversations | 12 | 5 | Handoff packets have recurring defects worth testing |
| Negative or repeated-question signal | 600 conversations | 10 | 6 | This queue is efficient for defect discovery, not prevalence |
| New shipping release | 1,200 conversations | 10 | 2 | Investigate both failures against the release stop condition |
| Low-volume languages | 180 conversations | 6 | 3 | Language coverage needs dedicated evidence despite low traffic |
The baseline's 3 defects in 30 reviews produce a point estimate of 10%, but the sample is too small for a confident weekly declaration that exactly 90% of conversations passed. Keep the raw numerator and denominator. After four comparable weeks, suppose the rolling baseline contains 120 reviews and 11 defects:
Rolling baseline defect estimate = 11 ÷ 120 = 9.2% Rolling baseline pass estimate = 1 − 0.092 = 90.8%
That rolling estimate is still subject to reviewer error and sampling uncertainty, but it is much more useful than treating one week's 30 cases as a precise score. Show the sample size beside every percentage.
The targeted half produces different value. Five defective handoffs can expose a missing account ID, absent escalation reason, or incorrect destination. Six negative-signal defects may reveal one repeated source gap rather than six independent product failures. Cluster them by cause before planning six fixes.
If leadership asks for a statistically narrower global estimate, increase the random baseline or review it less frequently in a larger batch. Do not steal cases from rare high-risk queues and pretend the operating risk disappeared. Estimation and discovery deserve separate budgets.
Review With a Fixed Record
Selection quality is wasted if reviewers apply different standards. Give every case a compact record:
conversation_id: conv_8421 sample_queue: new_shipping_release release: support-bot-2026-08-09.1 intent: shipping_cutoff expected_outcome: next-business-day processing after 14:00 observed_outcome: promised same-day processing at 14:18 grade: fail failure_layer: source_conflict evidence: old FAQ outranked current shipping policy customer_effect: false fulfillment promise fix_owner: content-operations regression_case: shipping-cutoff-after-1400
Define grades with observable behavior. “Helpful” and “felt wrong” are weak labels. Prefer grounded, complete, correctly routed, action confirmed, policy conflict, unsupported claim, missing handoff field, or duplicate effect. Save the source or system evidence that settled the grade.
Double-review a small portion of the set, including passes. If two reviewers disagree on 4 of 10 shared cases, the metric is measuring rubric ambiguity as much as chatbot quality. Resolve the examples, clarify the rule, and keep the disagreement rate as a QA signal.
Hide prior ratings when practical. A thumbs-down is a selection signal, not proof that the answer failed. The customer may dislike a correct policy; a cheerful rating can follow an incorrect discount promise.
Turn Samples Into Tests, Releases, and Routing Changes
A useful review ends with a controlled change or a documented decision to watch. Cluster defects by the layer that can repair them:
Source defects need a canonical page, a removed stale file, better parsing, or a reviewed Q&A pair.
Instruction defects need a narrower rule and examples of the boundary.
Routing defects need a corrected intent, priority, owner, or fallback destination.
Action defects need validation, confirmation, idempotency, or a tighter permission.
Coverage defects need more evidence before expansion, especially for rare languages and customer states.
Every material defect should create a regression case with the original phrasing, expected outcome, and evidence. The chatbot change-management process then provides the release ID, targeted tests, canary, stop condition, and rollback target for the fix.
Keep the random baseline stable long enough to see whether the change helped outside its targeted test set. A refund fix that passes ten refund prompts but increases unnecessary escalations elsewhere is not a clean win. Conversely, a targeted queue can stay intentionally hard; its pass rate may remain lower because the team keeps adding the newest failure modes.
Publish three lines in the weekly review:
- Baseline: numerator, denominator, trend window, and major uncertainty.
- Risk findings: defects by queue and customer consequence, without a fake global rate.
- Actions: owner, regression case, release or source change, and verification date.
That report is short enough to read and specific enough to operate.
Make the Review Budget Find Better Evidence
Chatbot QA sampling is a resource-allocation problem. Random review tells you how ordinary conversations are moving. Targeted review gives rare and consequential failures enough attention to become fixable. Keeping those queues separate preserves the meaning of both.
Start with a reproducible population, 20 to 30 random cases, and a targeted set sized around the risks your chatbot can create. Accumulate the baseline across comparable weeks, double-review enough cases to test the rubric, and turn each material defect into a regression test and owned change. More reviews help, but better selection makes every review hour produce stronger evidence.
In Agentkit, conversation logs and analytics provide the production review pool, while Q&A pairs can pin corrected answers that should not be improvised.
No credit card required.
