Sierra released Hyper-tau-bench on September 8 to test whether coding agents can build working customer-service agents from business records, client conversations, existing code, and production APIs. Its strongest autonomous setup passed 23.9% of held-out tasks. An expert-built reference using the same class of model reached 82.2%. The agent builders skimmed records, asked too few questions, and shipped the first design that ran.
Use this chatbot requirements checklist before selecting a model or configuring a prompt:
- Name one business job and its owner.
- List what the chatbot may answer, ask, do, and refuse.
- Map every source to an owner, effective date, and authority level.
- Record policy gaps that only a person can settle.
- Define each action's inputs, checks, side effects, and receipt.
- Set budgets for latency, cost, handoffs, and failures.
- Write acceptance tests from real conversations before development starts.
The deliverable is a small requirements packet, not a hundred-page specification. It should let a builder distinguish a missing rule from a software bug and let a reviewer decide whether the finished chatbot is safe to launch.
Start with one job and one accountable owner
"Handle customer support" is not a requirement. It hides dozens of decisions about refunds, identity, account access, product advice, exceptions, and escalation.
Write the job as an observable outcome:
Job: Answer pre-purchase questions about plans and integrations. Audience: Anonymous visitors on the pricing and product pages. Success: The visitor gets a sourced answer or reaches sales with useful context. Owner: Revenue operations. Out of scope: Account access, billing changes, refunds, and custom contract terms.
The owner must have authority to settle unclear rules. A project manager can collect requirements, but the billing owner decides refund policy and the security owner decides identity checks. Without named decision owners, developers fill gaps with plausible guesses. A fluent chatbot then turns those guesses into confident answers.
OpenAI's recent guide to turning agent workflows into repeatable operations recommends writing the agent's job description with its trigger, outcome, context, tools, permissions, evidence, and human review points. That is a useful minimum. For customer-facing work, add forbidden outcomes and the person who owns each rule.
Draw the boundary as customer situations
A list of features does not tell the chatbot when to stop. Write situations instead. Cover what happens when the answer is known, information is missing, sources conflict, the customer asks for an exception, identity is uncertain, or an external system fails.
For each situation, choose one allowed response mode:
- Answer. The chatbot has enough current evidence to give the requested information.
- Clarify. One missing fact, such as region or purchase date, changes the outcome.
- Collect. The chatbot may gather specified fields for a lead or support request.
- Act. It may call an approved tool after its required checks and confirmations.
- Hand off. A person must make the judgment or access a restricted system.
- Refuse. The request is outside scope or conflicts with a safety rule.
Do not use "use best judgment" for billing, eligibility, legal, medical, or account-security decisions. State the condition that changes the route. If nobody knows the condition, record a policy gap and keep the chatbot from making that decision.
The AI-to-human handoff guide can help turn the handoff mode into a specific destination, response expectation, and context packet.
Build a source map before uploading files
Support knowledge rarely arrives as one clean handbook. It lives in help pages, old PDFs, spreadsheets, ticket macros, release notes, and what experienced staff remember. Uploading all of it creates a larger search problem. It does not settle which source wins.
Create a source register with these fields:
| Field | Question it answers | Example |
|---|---|---|
source_id | What stable name will tests cite? | returns_policy_v7 |
scope | Which products, regions, and dates does it cover? | EU store orders after 2026-08-01 |
authority | Does it define policy or only describe past practice? | Approved policy |
owner | Who approves corrections? | Commerce operations |
effective_on | When does it start governing answers? | 2026-08-01 |
review_on | When must someone recheck it? | 2026-11-01 |
supersedes | Which older source must lose? | returns_policy_v6 |
Treat historical tickets as discovery material unless an owner approves the rule they appear to contain. A support agent may have made a one-time exception. Training a chatbot on that transcript can turn the exception into a public promise.
The practical setup in How to Train a Chatbot on Your Website Content covers crawling and document ingestion. Requirements work comes first because the crawler cannot infer ownership, authority, or expiry from a folder full of files.
Interview for rules that are not written down
Hyper-tau-bench exposed a blunt failure mode. In tasks where the client alone held 20 to 25 requirements, automated builders asked at most four questions. The research paper describes requirements spread across operating procedures, transcripts, images, audio, fee schedules, email, and human stakeholders. Keyword search found fragments, not the complete operating rule.
Ask domain owners questions that force a decision:
- Which customer request causes the most expensive support mistake?
- What exception do experienced agents know that the public page omits?
- Which two sources disagree today, and which one should win?
- What must the chatbot never promise, even if a customer insists?
- Which fact changes an answer from self-service to human review?
- What evidence must appear before anyone treats an action as complete?
Record the answer as a rule with an owner and date. Meeting notes alone are too soft. If the owner says, "We usually approve that," ask for the condition that makes approval valid and the condition that blocks it.
Specify actions as contracts
"Connect the chatbot to refunds" says almost nothing. A builder still has to guess who may request a refund, which fields are required, whether the tool issues money or opens a case, how duplicates are prevented, and what the user sees when the API times out.
Give every action a contract:
action: submit_refund_review allowed_for: authenticated_customer required_inputs: [order_id, reason_code] preconditions: - order belongs to authenticated customer - no open refund review exists for order confirmation: required before submission effect: create review case only receipt: case_id and expected response window on_timeout: check case status before retrying forbidden: promise approval or claim money was returned owner: billing_operations
The distinction between "open a review" and "issue a refund" is small in a feature brief and enormous to a customer. It should appear in the tool name, confirmation copy, success message, logs, and test cases. For higher-risk effects, use the approval patterns in Chatbot Approval Workflows.
Worked example: turn an ambiguous request into a testable rule
Suppose a retailer's FAQ says unopened goods can be returned within 30 days. A support macro says damaged goods can be reported within 60 days. A customer asks:
My headphones arrived damaged 45 days ago. Can you refund them today?
The original requirement reads, "The chatbot should help with returns." Three builders could reasonably produce three incompatible behaviors: deny the request after 30 days, promise a refund within 60 days, or send every return to support.
After interviewing commerce operations, the team learns that damaged items reported from day 31 through day 60 need manual review. The chatbot may collect the order number and damage description, but it cannot approve or issue money.
The requirement becomes:
case_id: damaged_item_day_45 known_facts: item_condition: damaged purchase_age_days: 45 required_outcome: - explain that the report is inside the 60-day damage window - state that support must review requests after day 30 - offer to collect order number and damage description forbidden_outcome: - deny solely because more than 30 days passed - promise approval or a completed refund allowed_action: submit_refund_review evidence: damaged_goods_policy_v3 owner: commerce_operations
A passing reply can use any clear wording. The test judges the decision, facts, and permitted action. It does not reward one memorized paragraph. The chatbot ground-truth guide shows how to maintain these records after launch.
Give cost and latency their own requirements
An accurate chatbot that takes 25 seconds to answer a checkout question is not ready. Neither is a cheap configuration that adds three clarification turns to every simple request.
Set separate limits instead of one blended score:
Median complete-answer latency: at most 4 seconds 95th-percentile complete-answer latency: at most 10 seconds Cost per 100 completed conversations: at most $12 Unnecessary handoff rate on common FAQ tests: below 5% Repeated tool side effects: zero Unsupported price or refund claims: zero
Use real conversation lengths and retrieval payloads when estimating cost. A one-turn demo understates the expense of clarification, retries, and tool calls. Keep hard safety limits separate from tradeoffs. Faster answers cannot compensate for an unauthorized refund promise.
Write acceptance tests before the first prompt
Requirements become useful when they can fail a build. Create tests while domain owners are still available, not after the team has tuned a chatbot around its own assumptions.
Start with 20 to 30 cases. Include common questions, paraphrases, missing facts, conflicting sources, policy exceptions, attempted prompt abuse, API failures, duplicate submissions, and a customer who changes their mind. Each case needs expected facts, forbidden claims, an allowed response mode, expected tool behavior, and governing evidence.
Keep some cases hidden from the builder. Visible tests guide development; held-out tests reveal whether the chatbot learned the rule or copied examples. Run the complete conversation, then inspect the final business state. A polite confirmation is not proof that a ticket or refund review exists.
Use the chatbot model evaluation scorecard after the requirements and test set are stable. Model selection comes late because a stronger model cannot repair an undecided policy or a tool with the wrong effect.
Version the packet and reopen it when evidence changes
Save the job definition, boundary rules, source register, decisions, action contracts, budgets, and acceptance cases under one release ID. Record open questions separately. An unresolved item should have an owner and due date, plus the safe behavior until it is answered.
Requirements do not freeze the business forever. They freeze the assumptions behind one build so the team can explain a result. Reopen the packet when a policy changes, a new source replaces an old one, an action gains authority, production logs reveal an unhandled case, or the chatbot moves to a new audience.
Hyper-tau-bench is a warning against confusing runnable software with a finished support system. The gap appears before code, when nobody has captured the rules the chatbot must follow. A compact, testable requirements packet gives builders something better than confidence: it gives them decisions they can implement and evidence reviewers can check.
In Agentkit, website pages, documents, text snippets, and reviewed Q&A pairs can supply the approved knowledge, while conversation logs show which requirements need another pass.
No credit card required.
