Chatbot Ground Truth: How to Build a Reliable Answer Key

OpenAI's July audit of SWE-Bench Pro estimated that roughly 30% of its evaluation tasks were broken. A separate field report on agentic scientific computing found agents could sound confident while producing clear errors; the strongest checks used external references or measurable acceptance targets. Google DeepMind describes the same pressure as a new validation bottleneck. Those examples concern code and science, but the operational lesson transfers directly to customer chatbots: a score is only as trustworthy as the answer key behind it.

Build every chatbot ground-truth item with six fields: the customer question, relevant facts, accepted outcome, forbidden claims, evidence, and review date. Add an owner whenever the answer depends on business policy. If a reviewer cannot fill those fields without guessing, the case is not ready to judge a chatbot.

This guide provides the record format, a source-precedence rule, a worked refund example, and a maintenance loop for keeping the answer key reliable as products and policies change.

Observed case study: knowing a plan did not mean recommending it

On September 7, 2026, we tested Agentkit's own public website chatbot with six pricing questions, twice each. The bot gave the correct Standard plan facts in both direct questions. In both fresh conversations asking for the cheapest plan with two chatbots and three team seats, it stopped short of recommending Standard and sent the visitor to the pricing page instead.

That is a narrower and more useful finding than an overall accuracy claim. A bot can state the correct facts when the question names the plan and still leave a buyer's related decision unfinished. This observation does not establish whether the cause was retrieval, instructions, a model, or another part of the system.

Download all 12 answers, the original answer key, source snapshot and grading decisions.

What we recorded before testing

We wrote six synthetic questions and their acceptance criteria before collecting answers. The governing source was the public Agentkit pricing page, checked that day with monthly billing selected. We judged the displayed plans' included limits, excluding add-ons and negotiated offers. For example, Standard listed $119.99 per month, 12,000 messages, two chatbots and three team seats.

We used the production widget on that page in Chrome 152.0.7977.83 on macOS 26.5.2. We ran each question once, then repeated the set in the same order. Between questions, we used the widget's reset control and checked that only the welcome message remained. We did not change the bot's configuration or submit feedback ratings.

An answer passed when it conveyed every required fact without a contradictory plan claim. Missing a required fact meant incomplete; a contradictory fact meant incorrect. We preserved all outputs, including the incomplete answers.

The decision question exposed a gap the fact question missed

The direct question asked:

What is Agentkit Standard's monthly-billed price, message allowance, chatbot limit and team-seat limit?

Both replies supplied the four correct facts. A separate question asked:

I need 2 chatbots and 3 team seats. What is the cheapest Agentkit plan that includes both, billed monthly, and what does it cost?

Both replies correctly ruled out Hobby because it includes one seat. Neither named Standard or its price. The first said that the available details did not specify the first tier meeting both requirements; the second also said it could not confirm without guessing. Both linked to the current pricing page.

Those replies avoided inventing an entitlement, but the requested decision remained unanswered. Marking them incomplete preserves that distinction. The direct lookup and decision cases should remain separate in a test set, even though they share source facts.

Our answer key also needed a correction

The API question asked whether Free included REST API access and which monthly-billed plan was cheapest if it did not. Both replies correctly said Free lacked API access and identified Hobby. Our prewritten rubric additionally demanded $29.99, although the question did not explicitly ask for the price.

After collection, we excluded both API outputs from the scored denominator rather than call that omission a chatbot failure. The downloadable record retains the original criterion, both answers and the exclusion reason. Of the ten remaining outputs, eight met the acceptance criteria and two were incomplete. This small example is not an estimate of production accuracy.

Before rerunning that case, decide which job you intend to test. If the price matters, ask for it explicitly. Keep the amended case version separate from this original record.

What this test can and cannot establish

Agentkit conducted this test on its own product using an automated assistant. The prompts are synthetic, and one automated reviewer graded them. We did not test Chatbase or compare models. The public widget did not expose its model ID, prompt version, retrieved context or release identifier. Repeated replies may share caching or retrieval behavior, so two runs do not establish a probability of failure.

The reusable contribution is the full question-and-answer record and the distinction it demonstrates: test the customer's decision as well as the individual facts, and audit the answer key before trusting its score. Recheck the source prices before repeating the study. For a consequential release decision, add representative customer cases, independent review and the configuration evidence missing from this public-widget test.

Define the Decision Before Writing the Expected Answer

Ground truth is not a polished sample response. A chatbot can phrase a correct answer several ways, and one memorized paragraph can become wrong when the customer, date, plan, or jurisdiction changes.

Start by deciding what a reviewer should prove. For a shipping question, the decision might be whether the chatbot used the current cutoff and avoided promising a delivery date the carrier has not confirmed. For a refund request, it might be whether the bot recognized an exception and escalated instead of inventing eligibility.

Use one record per decision boundary:

Keep style requirements separate from factual acceptance. “Uses the customer's name” may be a tone preference. “Does not claim the refund was issued” is an outcome rule. Mixing both into one pass/fail label makes a harmless wording difference look equivalent to a false financial promise.

The chatbot model-evaluation guide explains how to compare candidate models on a fixed test set. These ground-truth records make that set defensible before any model is scored.

Establish Which Source Wins

Most disputed test cases are not model mysteries. They expose two pages, people, or systems that disagree.

Write a source-precedence rule before adjudicating cases. A practical default is:

1. Live system state for customer-specific facts such as order status, entitlement, balance, or appointment availability.
2. Approved policy with an effective date for refunds, cancellations, warranties, eligibility, and compliance rules.
3. Maintained product documentation for features, setup steps, and supported behavior.
4. Reviewed Q&A entries for narrow answers that require exact wording or an intentional override.
5. Historical tickets and agent notes as discovery material, never as automatic authority.

Precedence alone is insufficient when the higher-ranked source is outside its scope. A billing system can prove that an invoice is unpaid; it may not define whether a fee should be waived. Store scope beside authority.

When two authoritative sources conflict, mark the item blocked_policy_conflict. Do not average the answers, choose the newest-looking page, or ask an evaluator model which policy seems more reasonable. Send the conflict to the named business owner and keep the case out of aggregate accuracy until the organization settles it.

If users need to see why an answer is trustworthy, the chatbot-citations guide shows how to return useful evidence without burying the response in links. The answer key should retain more detail than the customer-facing citation: source version, effective date, relevant passage or field, and retrieval timestamp.

Worked Case: One Refund Question, Three Plausible Answers

Suppose a customer writes:

I bought the annual plan 18 days ago and used about 300 messages. Can I get a refund?

The support site contains three pieces of evidence:

• a pricing FAQ last reviewed six months ago says annual plans have a 30-day money-back guarantee;
• the approved refund policy effective August 1 says refunds are available within 14 days;
• the billing system shows the purchase was 18 days ago, but the account is in a region with a mandatory 30-day withdrawal right.

A brittle answer key might say, “No, the purchase is outside 14 days.” That expected answer matches the general policy while missing the jurisdictional exception. Another reviewer might follow the old FAQ and accept an unconditional refund. Neither judgment is safe.

The resolved record should look like this:

case_id: refund_annual_018
question: "I bought the annual plan 18 days ago and used about 300 messages. Can I get a refund?"
facts:
  purchase_age_days: 18
  plan: annual
  usage_messages: 307
  customer_region: region_with_30_day_withdrawal
accepted_outcome:
  - state that the request needs the regional withdrawal rule
  - avoid denying eligibility based only on the general 14-day policy
  - route to billing review with purchase date and region
forbidden_claims:
  - "You are not eligible because 18 days is outside 14 days"
  - "Your refund has been issued"
evidence:
  - source: approved_refund_policy_v4
    effective: 2026-08-01
    scope: general
  - source: regional_withdrawal_schedule
    effective: 2026-01-01
    scope: customer_region
owner: billing_policy
review_on: 2026-11-01

Now evaluate three candidate replies:

Reply A: “Refunds are only available for 14 days, so this purchase is not eligible.” Fail: it ignores the regional exception and makes a forbidden denial.

Reply B: “You are within the 30-day guarantee, so I have issued the refund.” Fail: the old FAQ is not canonical, and no refund action occurred.

Reply C: “Your location may give you a 30-day withdrawal period. I’ll send billing your purchase date and region to confirm the request; no refund has been issued yet.” Pass: it preserves the applicable exception, takes only the allowed action, and describes the current state accurately.

Notice that the passing response does not copy a golden paragraph. It satisfies observable conditions. That makes the case useful across prompt changes and models without rewarding one writing style.

Test Variations Without Changing the Truth

One case rarely represents one user sentence. Create variations that preserve the same decision while changing phrasing, order, and irrelevant detail:

• “Am I too late to cancel my yearly subscription? I signed up 18 days ago.”
• “Your FAQ says 30 days, but another page says 14. Which applies to me?”
• “I barely used it. Refund my annual plan.”
• “The charge was on July 25 and I live in this region. Can support reverse it?”

Then create boundary cases that intentionally change the expected outcome: day 13 under the general policy, day 31 in the protected region, a monthly plan, a completed refund, and a customer whose region is unknown. Label which fact caused the answer to change.

Avoid generating dozens of paraphrases before the base record is reviewed. Synthetic variety can multiply an unnoticed policy error until the test set looks large and independent. Approve the decision record first, then expand wording.

Also separate answer tests from action tests. A reply may explain eligibility correctly while calling a refund tool without confirmation. Record the expected tool behavior, side-effect count, and receipt when the workflow can act. Conversational correctness cannot prove that a backend action was authorized or completed.

Adjudicate Disagreement Instead of Voting It Away

Review a small shared batch with two people before using a new answer key for release decisions. Give each reviewer the same facts and evidence, but hide the other reviewer's label.

Classify disagreements by cause:

• Missing fact: the case omitted a region, plan, date, identity state, or prior action needed to decide.
• Ambiguous acceptance rule: reviewers agree on the policy but interpret “complete” or “safe” differently.
• Source conflict: evidence supports incompatible outcomes.
• Reviewer mistake: one label contradicts the stated evidence.
• Policy gap: the organization has never decided the edge case.

Do not resolve every split with majority vote. Voting can hide a policy gap behind a number. A domain owner should settle source and policy conflicts; the evaluation owner should rewrite ambiguous cases; simple reviewer mistakes should retain a short correction note.

Track agreement on the outcome fields, not whether reviewers wrote the same ideal response. If 6 of 20 shared cases produce different pass/fail labels, pause scoring and repair the rubric. A leaderboard built on unresolved labels rewards noise.

Production review can supply the next cases. Use the random and risk queues in the chatbot QA sampling plan, then promote material failures into ground-truth records only after confirming the evidence. A customer thumbs-down is a useful discovery signal, not an answer key.

Measure the Answer Key as Its Own Product

Report chatbot performance and ground-truth health separately. At minimum, track:

Coverage = eligible test cases with approved ground truth / all eligible test cases
Freshness = approved items reviewed before expiry / all approved items
Agreement = shared items with matching reviewer outcomes / all shared items
Blocked rate = policy-conflict items / all drafted items

Suppose a release set contains 120 drafted cases. Ten are blocked by policy conflicts, 90 have approved records, and 20 are missing evidence or ownership. Of the 90 approved records, 81 are still within their review window. Two reviewers independently label 20 cases and agree on 18.

Coverage = 90 / 110 eligible cases = 81.8%
Freshness = 81 / 90 = 90.0%
Agreement = 18 / 20 = 90.0%
Blocked rate = 10 / 120 = 8.3%

The chatbot may score 95% on the 90 approved cases, but that number does not erase the missing 20 or conflicted 10. Report both layers: model or system quality on approved truth, and the completeness of the truth set itself.

Set review triggers by volatility. Pricing and promotion records may expire on a launch date. Compliance answers should be reviewed when the controlling rule changes. Stable setup instructions can use a longer calendar interval. Any source change, repeated production dispute, or owner change should trigger early review.

Archive superseded records rather than editing history in place. Keep the prior evidence, dates, judgment, and evaluation runs tied to that version. Otherwise yesterday's score may silently acquire today's answer key and become impossible to reproduce.

Make Correctness Reviewable

A reliable chatbot answer key records decisions, evidence, boundaries, and ownership. It accepts multiple good phrasings while rejecting specific harmful claims. It also admits when the organization has not supplied enough truth to judge the model.

Start with 20 high-volume or high-consequence questions. Give each one the six required fields, double-review a sample, and resolve policy conflicts before publishing an accuracy score. Expand only after the records are current and reviewers can reach the same outcome from the same evidence.

In Agentkit, Chat Logs expose real customer questions and source references, while reviewed Q&A pairs can override future responses for critical answers that need exact control.

Build your chatbot for free →

No credit card required.