OpenAI introduced on-demand tool search with its new Agents API on September 10. Instead of putting every tool definition into every prompt, the agent can load relevant tools when it needs them. That saves context, but it also puts a sharper question in front of builders: can your chatbot find the right action from the catalog you gave it?
Start with this quick audit. For every tool, check that its name identifies the system and job, its description says when to use it and when not to, its inputs match the user's language, its output proves what happened, and its nearest competitor is covered by a contrast test. Then confirm that a no-action request returns no tool call. A catalog that fails any of those checks needs work before you add another integration.
Selection has two failure points
Tool selection is often treated as one model decision. In a large catalog, it is two decisions:
- A search or routing layer chooses a small set of candidate tools.
- The model chooses one tool, several tools, or no tool from that set.
Those steps fail differently. If the correct tool never reaches the shortlist, rewriting the final prompt will not fix the problem. If the shortlist contains the correct tool and the model still picks a neighbor, the names, descriptions, examples, or request context are probably ambiguous.
Keep separate measurements for the two stages:
| Metric | Formula | What it diagnoses |
|---|---|---|
| Shortlist recall | Requests where the correct tool appears divided by eligible requests | Search and routing quality |
| Selection accuracy | Correct choices divided by requests where the correct tool appeared | Model choice among candidates |
| No-tool accuracy | Correct no-call decisions divided by requests that need no action | Unnecessary action risk |
| End-to-end success | Verified outcomes divided by all eligible requests | The whole path, including execution |
A single success rate hides where the repair belongs. Suppose 100 requests include 80 that need a tool. The router includes the right tool for 72, and the model selects it for 63. Shortlist recall is 90%, while selection accuracy is 87.5%. Improving the model cannot recover the eight requests that lost the correct tool during retrieval.
The distinction becomes more important as catalogs grow. A May 2026 study evaluated registries with up to 3,251 tools. On one 370-tool benchmark, an adaptive policy showed the model about seven tools on average while nearly matching the coverage of a fixed 50-tool list. Its downstream validation also found better model selection with shorter adaptive lists. The paper separates retrieval, choice, and execution, which is the useful operational lesson even if you never copy its method.
Give every tool one recognizable job
A REST API describes what software can call. A chatbot tool must describe a job the model can recognize from a customer's words. Copying every endpoint into the tool catalog usually creates overlapping choices with implementation-shaped names.
Consider an order API with these endpoints:
get_orderget_order_statussearch_ordersupdate_ordercancel_ordercreate_support_case
The first three may all look plausible when a customer asks, "Where is order 7412?" The broad update_order tool may cover cancellation, address changes, and delivery instructions, each with different confirmation rules. The model must infer product architecture before it can help the customer.
Design around customer jobs instead:
| Tool | Use when | Do not use when | Required proof |
|---|---|---|---|
store_order_track | The customer asks where an existing order is or when it will arrive | The customer wants to change or cancel it | Current carrier status and checked-at time |
store_order_cancel_request | The customer explicitly asks to cancel an eligible order | The customer asks only about status, returns, or address changes | Cancellation request ID and state |
store_order_change_address_request | The customer confirms a new delivery address before shipment | The order has shipped or identity is unverified | Change request ID and accepted address summary |
support_case_create | The request needs a human or no order tool can complete it | A read-only answer or safe action can finish the job | Case ID and queue |
The service prefix tells the model where the action lands. The job name separates read from write. The description draws a boundary around nearby tools. The proof field gives the model evidence for its reply.
Anthropic's guide to writing tools for agents recommends a few workflow-shaped tools instead of wrappers around every low-level endpoint. It also notes that namespacing can help models distinguish overlapping tools. Treat that as a testable design choice, not a naming fashion. The right prefix depends on the models and requests you actually use.
A worked selection trace
Here is a realistic request:
Customer: "Order 7412 still says confirmed. Did it ship?"
The router should retrieve store_order_track and perhaps support_case_create. It should not retrieve either write tool. The model selects the tracking tool because the customer asked for state, not a change.
{
"tool": "store_order_track",
"arguments": {
"order_reference": "7412"
}
}
The tool returns:
{
"order_reference": "7412",
"fulfillment_state": "packed",
"carrier_state": null,
"checked_at": "2026-09-13T08:42:11Z",
"next_step": "wait_for_carrier_scan"
}
A good reply says the order is packed but has no carrier scan yet. It does not claim that it shipped. This is where selection meets action verification: choosing the right tool begins the job, while interpreting its result determines what the chatbot may claim.
Now change one word:
Customer: "Order 7412 still says confirmed. Cancel it."
The router should retrieve store_order_cancel_request, plus any identity or eligibility check that the runtime requires. The model should ask for explicit confirmation if none exists. store_order_track can provide useful context, but it cannot satisfy the request.
This paired example belongs in your permanent test set. Minimal contrasts expose fuzzy tool boundaries faster than a long collection of unrelated happy paths.
Build contrast tests before adding more tools
Start with real customer wording from chat logs. Remove personal data, label the expected tool or no-tool result, and group requests by the tools most likely to be confused. Include incomplete, indirect, and conversational phrasing. Customers rarely repeat your schema description.
Use a table like this for every cluster of neighboring actions:
| Test prompt | Expected decision | Dangerous alternative |
|---|---|---|
| "Has 7412 left the warehouse?" | store_order_track | store_order_cancel_request |
| "Stop 7412 before it ships" | Ask for confirmation, then store_order_cancel_request | store_order_track only |
| "Send 7412 to my office instead" | Verify identity and eligibility, then store_order_change_address_request | Broad order update |
| "What is your cancellation policy?" | No tool, answer from approved knowledge | Any order write tool |
| "It is too late to change the address, help" | support_case_create | Retrying the blocked address tool |
| "Cancel it and open a case if that fails" | Ordered multi-tool plan | Calling both tools at once |
Run each prompt several times against every supported model and reasoning setting. Record which candidates the router returned, which tool the model chose, the proposed arguments, and whether the runtime executed it. Variable outputs matter. A tool that passes nine times and cancels the wrong order on the tenth is not 90% ready.
Add three kinds of negative case. No-action cases ask for policy, explanation, or comparison. Missing-detail cases need a question before a call. Forbidden cases request an action that permissions or policy should block. Selection testing does not replace the tool permissions checklist, but it should confirm that restricted tools do not appear for ineligible users in the first place.
Keep the shortlist small without making it rigid
OpenAI says its Agents API tool search loads relevant definitions as needed, which reduces token use and helps preserve prompt caching. Anthropic reported a similar result in its advanced tool use release: a test configuration fell from about 77,000 tokens loaded before work began to about 8,700, and tool-selection scores improved in its internal evaluations.
The obvious mistake is replacing "show everything" with one fixed shortlist size. Easy requests may need one or two choices. A request that spans a CRM, billing system, and help desk may need more. Set an initial range, then tune it with shortlist recall and confusion data.
A practical routing policy can stay simple:
- Always include the no-tool option.
- Filter tools by user, tenant, channel, and permission before semantic ranking.
- Retrieve a small set for one-job requests.
- Expand the set when scores are close or the request clearly contains several jobs.
- Keep high-risk write tools out unless the request states the matching intent.
- Require confirmation after selection for consequential writes.
Search is not authorization. A perfect semantic match must still pass the runtime controls described in the AI agent hooks guide. Retrieval decides what the model can consider. Policy decides what may execute.
Treat no tool as a first-class result
Many catalogs optimize only for picking the correct tool when a tool is required. Website chatbots also answer questions, clarify intent, refuse requests, and hand work to people. Forcing every message into an action creates needless calls and strange behavior.
Write explicit no-tool examples:
- greetings and thanks;
- questions answered by approved content;
- requests missing an order, account, date, or destination;
- hypothetical questions about what an action would do;
- attempts to trigger internal or unauthorized tools;
- corrections that do not yet restate the desired action.
Track unnecessary-call rate by tool. A cancellation tool with zero failed executions can still be dangerous if the model selects it for policy questions and a later layer happens to block every call. Those blocks are saved incidents, not clean selections.
Ambiguity deserves its own result too. If crm_contact_update and billing_contact_update both score highly for "change my email," the chatbot should ask which email the customer means. Hiding uncertainty behind a confident tool call is worse than one short question.
Review the catalog from production evidence
Tool catalogs drift. Teams add aliases, vendors change schemas, descriptions accumulate exceptions, and customer language shifts. Version the catalog with the same care as the code that executes it.
Review these events every week:
- wrong-tool blocks from policy checks;
- queries where the expected tool missed the shortlist;
- repeated clarification loops;
- tools retrieved often but selected rarely;
- tools selected often but completed rarely;
- new phrases that operators use when correcting the chatbot.
Every repair should land as a catalog change plus a regression test. Rename an ambiguous tool, tighten its boundary, add a contrast example, or split a job only when the evidence points there. Do not keep stacking prompt warnings around a bad catalog.
Agentkit supports custom API calls, forms, buttons, and lead capture actions. Its conversation logs give you the raw material to find customer wording and review failures. Begin with the two actions people confuse most, write ten paired tests, and fix the names before you connect another system.
The best tool catalog is not the largest one. It is the one where the correct action is easy to find, the wrong action is easy to reject, and doing nothing remains a valid choice.
No credit card required.
