AI model routing moved from an architecture discussion to a product category this week. On August 19, Ramp launched Router, a service that selects models by quality and cost. The same day, Stripe announced an agreement to acquire OpenRouter, which routes usage across hundreds of models and providers. The market signal is clear: production teams no longer want every request tied to one model.
For a chatbot team, the useful question is narrower: which requests may use a cheaper model, which require a stronger one, and what happens when the router is wrong? Start with five controls: define traffic classes, give each class a quality floor, route to the cheapest model that clears it, escalate on explicit signals, and preserve a safe fallback. The routing decision table below is the asset to build first.
Write a Five-Field Routing Policy
A router should execute a policy your support, product, and engineering teams can read. Do not start with a clever classifier and reconstruct its rules after an incident.
| Policy field | Decision to record | Support example |
|---|---|---|
| Traffic class | What kind of job is this? | FAQ lookup, account action, policy exception |
| Quality floor | What must always be correct? | Price matches the current billing page |
| Preferred tier | Cheapest tested tier that clears the floor | Fast, standard, or high-reliability model |
| Escalation signal | What moves the request upward? | Conflicting sources, missing identity, exception request |
| Failure path | What happens if no tier is eligible? | Clarify, hand off, or return a bounded answer |
Keep the policy independent of vendor names. “Standard tier” survives a model release; “always use Model X” becomes stale as soon as prices, latency, or behavior change. Map current models to tiers in a separate registry that can be evaluated and updated without rewriting routing logic.
This separation also makes ownership clearer. Support owns the traffic classes and unacceptable mistakes. Engineering owns execution, timeouts, and observability. The evaluation set decides which models are eligible. Procurement can compare providers without quietly lowering the answer-quality bar.
Route by Consequence, Not Prompt Length
Long prompts are not automatically hard, and short prompts are not automatically safe. “Summarize this public help article” may contain thousands of tokens but little decision risk. “Can you make an exception?” is short but could trigger an unauthorized discount, refund, or promise.
Use the consequence of a wrong answer as the first routing dimension. Then consider reasoning difficulty, context size, tool use, latency, language, and privacy.
| Traffic class | Typical requests | Minimum route | Escalate when | Safe failure path |
|---|---|---|---|---|
| Deterministic | Greetings, button labels, known status text | No model or smallest eligible tier | The request contains another intent | Show fixed copy or ask one question |
| Grounded lookup | Hours, shipping policy, feature availability | Fast tier with retrieval | Sources conflict or lack the answer | Cite the source or say it is unavailable |
| Comparative judgment | Plan comparison, troubleshooting, multi-document synthesis | Standard tier | Constraints conflict or evidence is incomplete | Explain the uncertainty and clarify |
| Consequential decision | Refund exception, access change, legal or privacy issue | High-reliability tier plus server checks | Policy is ambiguous or action needs approval | Hand off without claiming completion |
This table is a starting point, not a model benchmark. A fast model earns a class only after it passes that class’s evaluation set. The process in the chatbot model evaluation guide keeps prompts, retrieval, and grading constant so a routing decision is based on your workload rather than a public leaderboard.
Do not route solely on the first user message. Conversation state can change the class. “Where is my order?” begins as a grounded lookup. “The courier marked it delivered, but the photo is not my building” becomes a dispute with identity, evidence, and policy implications. Reclassify after tool results and after any turn that changes the requested outcome.
Choose the Smallest Router You Can Audit
There are three common designs, and most website chatbots should combine the first two.
Rules handle known boundaries. Authenticated actions, regulated topics, refund exceptions, data deletion, and requests containing sensitive fields can go directly to a restricted route. Rules are cheap, reproducible, and easy to explain during an incident. They are brittle when asked to understand every variation of ordinary language.
A classifier handles ordinary variation. A small model or purpose-built classifier assigns a traffic class and confidence. It can recognize that “my package never turned up” and “delivery says complete, nothing here” describe the same job. Limit its output to your known classes and validate the result before using it.
A cascade lets output quality trigger escalation. The cheaper eligible model answers first. A validator checks required facts, citations, tool results, or a narrow rubric. If the answer fails, the system retries on a stronger tier. Cascades are useful when difficult requests are hard to identify before generation, but they add latency and can pay for two calls.
A practical hybrid looks like this: hard rules reserve consequential work; a classifier routes the remaining traffic; deterministic checks inspect the answer; and only ambiguous failures enter a cascade. Avoid asking one language model to route, answer, and declare its own answer correct. That design turns a single blind spot into three agreeing decisions.
Worked Example: 10,000 Support Conversations
Suppose a support chatbot handles 10,000 conversations per month. Its current high-reliability model averages $0.08 per conversation, so model cost is $800. These are hypothetical unit costs; replace them with your invoices and measured token usage.
A replay test finds that 70% of conversations clear every hard gate on a fast tier at $0.003 each, 25% need a standard tier at $0.015, and 5% require the high-reliability tier at $0.08. Classification and validation add an average of $0.001 per conversation.
- Fast tier:
7,000 × $0.003 = $21.00 - Standard tier:
2,500 × $0.015 = $37.50 - High-reliability tier:
500 × $0.08 = $40.00 - Routing and validation:
10,000 × $0.001 = $10.00 - Total routed cost:
$108.50
The modeled saving is $800 - $108.50 = $691.50, or 86.4%. That number is not yet a business result. If 120 conversations are wrongly sent to the fast tier and 30 reopen as support tickets, the router may have traded inference cost for more expensive human work and angrier customers.
Calculate cost per durable resolution instead:
(model cost + routing cost + review cost + reopened-case cost) / conversations resolved without reopening
This denominator prevents a cheap but unreliable route from winning. The chatbot cost guide explains the other expenses that sit around inference and should be included in the same calculation.
Record a Route Receipt for Every Decision
When a routed answer fails, the team needs more than the final model name. Store a compact receipt with the conversation event:
{
"policyVersion": "support-router-7",
"trafficClass": "grounded_lookup",
"signals": ["shipping_intent", "source_found"],
"preferredTier": "fast",
"selectedModel": "provider/model-version",
"validator": {
"grounded": true,
"requiredFactsPresent": true
},
"attempt": 1,
"fallbackUsed": false
}
Do not log raw sensitive prompts merely to explain routing. Stable class labels, rule matches, document identifiers, model versions, latency, token counts, validator results, and redacted error codes are usually enough. Keep the policy version so an old conversation can be replayed against the rules that actually handled it.
The receipt should distinguish three events: routing miss (the wrong class was chosen), model miss (the class was right but the selected model failed), and system miss (retrieval, a tool, or policy data failed). Without that split, teams often upgrade the model to fix a stale knowledge source or change the classifier to fix a broken API.
Test Router Regret, Not Just Router Accuracy
Classification accuracy treats every mistake as equal. Routing regret measures the consequence of the decision compared with the best eligible route.
A needless upgrade from fast to standard has small cost regret. Sending a billing exception to a model that invents an approval has severe quality regret. Track them separately rather than averaging them into one comforting percentage.
Your weekly routing review should include:
- Hard-gate failure rate by route: unsupported claims, missed approvals, privacy failures, or incorrect actions;
- Unnecessary upgrade rate: requests that a cheaper eligible tier would have handled;
- Late escalation rate: conversations that reached a stronger tier only after a bad answer was shown;
- Fallback success rate: requests completed safely after a provider timeout or refusal;
- Durable resolution and reopening rate: outcomes after 24 or 72 hours, not just first-session completion;
- Latency and cost by traffic class: averages plus the slowest five percent.
Sample routed conversations from every class, including successful ones. Reviewing only escalations teaches you about obvious failures but not the expensive route that quietly receives easy traffic. If you need a confidence threshold for answer-versus-escalate decisions, use the calibration process in the chatbot confidence score guide rather than treating the model’s verbal confidence as a probability.
Make Fallbacks Preserve the Quality Floor
Provider outages and rate limits are when routing policy is easiest to violate. A fallback should be another model proven for the same traffic class, not whichever endpoint still responds.
For grounded lookups, a peer model may be safe if it receives the same retrieved evidence and passes the same citation checks. For a consequential decision, a weaker model should not inherit authority because the preferred provider timed out. Return a bounded status, collect the necessary details, or hand the case to a person.
Set retry budgets by user experience. One fast retry may be invisible; three sequential model attempts can turn a support widget into a loading screen. Run independent provider health checks outside the customer request so the router does not discover a broad outage one conversation at a time.
Version the fallback tree and test it deliberately. Simulate timeouts, malformed output, quota errors, missing retrieval results, and a model that responds successfully but violates a hard gate. The model rollback checklist is useful when an upstream model changes behavior even though its API name stays stable.
Roll Out With Shadow Decisions First
Before a router controls production, run it in shadow mode. The current model still answers customers while the router records which tier it would have selected. Replay the same redacted conversations through candidate tiers and grade them against class-specific gates.
Move one low-consequence class at a time. Start with grounded FAQs that have authoritative sources and an obvious refusal when evidence is missing. Hold out a control group, compare durable resolution and cost, and stop the rollout if any hard-gate failure rises. Only then add comparative or action-adjacent traffic.
Set an expiration date on routing evidence. A model that passed a shipping-policy set three months ago has not automatically passed today’s policy, prompt, retrieval index, or provider revision. Re-run the class evaluation after any of those inputs changes.
AI model routing is valuable when it makes cost a constrained optimization problem: spend less only after answer quality, safety, and recovery requirements are met. The router’s real product is not model choice. It is an auditable promise that each request receives no less capability than its consequences demand.
No credit card required.
