Anthropic's Claude Opus 5.5 release on September 22 made a quiet production detail impossible to ignore. Some requests can move to another model when a safety classifier intervenes. The requested model ID may stay in your configuration while a different model writes the answer.
Before enabling any automatic fallback, require four proofs: your logs name every attempted and serving model, the fallback passes the same policy tests as the primary, a refusal cannot masquerade as a normal success, and the extra attempt stays inside your latency and cost limits. The test matrix below turns those requirements into a release gate.
Start with a fallback contract
A fallback is a routing decision, not an error handler. Write down when it may happen, which models may serve, what the customer sees, and which event your monitoring receives. If any field is unknown, the route is not ready for production.
| Field | Required value | Why it matters |
|---|---|---|
requested_model | Exact model ID sent by your app | Proves what the application intended to use |
served_model | Exact model ID that produced the answer | Reveals a provider-side or application-side switch |
trigger | Refusal category, timeout, overload, rate limit, or manual route | Separates safety behavior from availability failures |
attempts | Ordered list with status for every model call | Exposes loops and repeated charges |
usage_by_attempt | Input, output, cache, and billed tokens | Prevents the final response from hiding total cost |
first_byte_ms and total_ms | Customer-facing latency for the whole chain | Captures time spent on the failed first attempt |
conversation_route | Primary, fallback, or returned-to-primary | Makes multi-turn drift visible |
user_outcome | Answered, refused, escalated, or failed | Keeps a technical retry separate from a solved request |
Do not infer these fields from response text. A polite answer can come from the fallback. A refusal can arrive as a successful HTTP response. Record the routing metadata at the boundary where your application receives the provider response.
HTTP 200 can contain a failed first attempt
Anthropic's current refusal and fallback documentation is a useful concrete example. A classifier refusal returns HTTP 200 with stop_reason: "refusal". With server-side fallback enabled, the API can retry the same request on another model and return one response. The top-level model, a fallback content block, and usage.iterations show what happened.
Your application event should make that chain obvious:
{
"conversation_id": "conv_81f4",
"turn_id": "turn_12",
"requested_model": "primary-model",
"served_model": "fallback-model",
"trigger": "classifier_refusal:cyber",
"attempts": [
{
"model": "primary-model",
"result": "refusal",
"input_tokens": 412,
"output_tokens": 0
},
{
"model": "fallback-model",
"result": "answered",
"input_tokens": 412,
"output_tokens": 264
}
],
"first_byte_ms": 1840,
"total_ms": 3260,
"user_outcome": "answered"
}
This is an application-level event, not a vendor response copied verbatim. It gives support and engineering one record they can query across providers.
The distinction matters during streaming. A refusal may happen before output or after partial output. Vendor behavior can differ between streaming and non-streaming requests, including whether partial text carries into the second attempt. Test both modes. Never stitch partial output from two models together unless the provider protocol explicitly defines that behavior and your renderer understands the boundary.
Run six tests against the whole chain
Use fixed prompts and expected outcomes. Run each case several times because classifier and model behavior can vary. Keep the primary prompt, tools, sources, and customer state identical across runs.
| Test | Forced condition | Pass condition |
|---|---|---|
| Primary success | Ordinary supported question | One attempt; requested and served models match |
| Eligible refusal | Benign prompt known to reach an approved fallback category | Refusal event recorded; approved fallback answers; model switch visible |
| Final refusal | Prompt that policy says must remain blocked | No fallback bypass; user gets the approved refusal or human handoff |
| Fallback unavailable | Rate-limit or disable the fallback in staging | One bounded failure; no retry loop; clear escalation path |
| Mid-stream refusal | Trigger intervention after partial output in a test environment | UI discards or handles partial text exactly as designed |
| Next conversation turn | Continue after a fallback-served answer | Route is explicit; history remains valid; tools and citations still work |
That last test catches a subtle failure. Some systems keep a conversation on the fallback model for later turns. If your dashboard labels every later message with the configured primary, reviewers will blame the wrong model for changes in tone, tool selection, or refusal behavior.
The test set should include the cases most likely to trip safeguards accidentally. For a support chatbot, that may include security questions, account recovery, health-related product questions, regulated claims, and customers quoting suspicious text from an email. Add real false refusals from conversation review. The chatbot refusal testing guide shows how to separate missing evidence, unclear scope, and excessive caution before changing the route.
Decide which refusals may fall through
Automatic fallback can improve availability, but a refusal is not the same as a timeout. A second model may answer because its policy, classifier, or capability boundary differs. Treating every refusal as a transient failure can undo the control that produced it.
Approve fallback by category and customer outcome:
- A public documentation question falsely classified as risky may use a qualified fallback if both models have passed the same answer and policy tests.
- An account-recovery request should usually escalate when identity evidence is missing. Another model must not guess its way around that boundary.
- A request for a prohibited action should remain refused. Retrying until one model complies is policy shopping.
- A tool call that can charge, delete, publish, or expose private data needs the same deterministic authorization checks on every route.
Named fallback models are easier to qualify than a provider-managed default that can change. A managed route may still be appropriate, but pin your acceptance criteria to observable behavior rather than an assumed model name. Alert when the serving model falls outside the set you tested.
This is where the broader chatbot model evaluation scorecard remains useful. Apply its hard gates to the fallback separately. Do not let a high average score offset one unauthorized action or policy contradiction.
Measure the chain, not the final response
Track four rates together:
primary refusal rate = primary refusals / eligible turns fallback serve rate = fallback answers / primary refusals stranded refusal rate = unanswered or failed turns / primary refusals fallback regression rate = failed fallback grades / fallback answers graded
Then segment each rate by refusal category, intent, language, channel, customer tier, requested model, and served model. An overall fallback rate can look stable while one account-security intent jumps from 1% to 14%.
Latency also belongs to the full chain. Measure time to first byte and completion from the customer's original request, not from the fallback attempt. A retry that produces an answer in 900 milliseconds after a three-second refusal still took nearly four seconds.
Cost needs the same treatment. Keep usage by attempt and calculate cost per answered conversation. OpenAI's recent customer-support optimization cookbook recommends treating policy compliance, action correctness, security, and escalation accuracy as hard gates before comparing cost or latency. That ordering is right for fallbacks too. A cheaper recovered answer is not a saving if it creates a second contact or an incorrect promise.
A worked fallback drill
Imagine a support team replays 10,000 redacted production-shaped turns in staging. The primary model refuses 180 turns. The fallback answers 162 and leaves 18 unanswered.
At first glance, a 90% fallback serve rate looks strong. The graded results tell a different story:
Public product questions: 118 fallback answers, 118 pass Security education: 29 fallback answers, 28 pass, 1 vague Account recovery: 15 fallback answers, 12 pass, 3 critical failures Primary-only p95 latency: 2.1 seconds Fallback-chain p95: 4.8 seconds Fallback loops: 0 Unlogged model switches: 0
All three critical failures skipped a required identity check and offered account-specific next steps. The team should not average those failures across 162 answers and ship the route.
The release note becomes:
decision: limited_release allow_fallback: - public_product_questions - security_education_without_account_data block_fallback: - account_recovery on_blocked_refusal: escalate_to_human stop_conditions: - any missing identity check - any unapproved serving model - fallback-chain p95 above 5 seconds review_after: 500 fallback-eligible production turns
That artifact states what can switch, what must stop, and when the team will review real traffic. It also leaves a clean rollback path. If the provider changes its default route or the fallback begins failing a hard gate, disable the route and return to the last qualified behavior. The model rollback checklist covers the wider incident record and recovery decision.
Choose where the routing lives
Provider-side fallback reduces client code and may save a network round trip. It can also carry provider-specific response blocks, billing rules, sticky routing, and beta headers. Your application still owns the logs, release gate, and customer outcome.
Application-side fallback gives you direct control over the model list, category rules, retry budget, and cross-provider routing. It also makes you responsible for preserving valid conversation history, preventing duplicate tool effects, handling cache charges, and keeping every retry path in sync.
Do not combine safety fallback, outage failover, and cost routing under one generic retry label. They answer different questions:
- Safety fallback asks whether another approved model may answer after a classified refusal.
- Availability failover asks whether another service may answer after a timeout, rate limit, overload, or server error.
- Cost routing chooses a qualified model before the first attempt.
Amazon Bedrock's intelligent prompt routing documentation, for example, describes selecting among models and configuring a fallback model when routing criteria are not met. That is a different control from retrying a safety refusal. Separate events and thresholds keep the distinction visible.
Make the switch visible before customers see it
A model fallback can rescue a conversation, but only if the second route is tested as a first-class production path. Log the requested and serving models. Preserve attempt-level usage. Grade the fallback on the intents that trigger it. Keep refusals that protect customers from unsafe or unauthorized actions intact.
Anthropic's evaluation guidance notes that regression suites should answer a strict question: does the agent still handle everything it used to? The same principle applies here. The fallback earns production traffic when it preserves required behavior under the exact condition that removed the primary model from the turn.
No credit card required.
