Long-running agents create more chances for a small failure to become a repeated one. In July, OpenAI described how persistence gives autonomous models more opportunities to take unwanted actions and argued for monitoring that can intervene, pause, or roll back. A day later, its disclosure about an agent-driven security incident showed why runtime containment cannot depend on a model deciding when to stop.
Production AI products are moving in the same direction: more connected systems, approved actions, policies, and escalation paths. OpenAI's description of production voice and chat agents makes those operating controls explicit. The practical control to add now is an AI agent circuit breaker. It watches outcomes outside the model, blocks a failing action path, serves a safe fallback, and permits a limited recovery test only after the destination appears healthy.
Start With a Circuit-Breaker Trip Card
Do not begin with a generic instruction such as “stop if too many calls fail.” Give every consequential tool a small, reviewable trip card. This starter version is useful enough to implement and specific enough to test:
| Signal | Starter threshold | Breaker response |
|---|---|---|
| Definite failures | 5 failures in 20 attempts | Open the affected tool for 10 minutes |
| Unknown outcomes | 2 timeouts after submission | Open writes immediately; start reconciliation |
| Authorization denials | 3 denials for one actor in 5 minutes | Block that actor and alert security |
| Rate-limit responses | 3 responses in 60 seconds | Pause calls for the provider's retry window |
| Step count | More than 8 tool calls in one run | Stop the run and route to review |
| Spend or token budget | 100% of the per-run limit | End the run without starting another tool call |
| Recovery probes | 4 consecutive verified successes | Close gradually; keep reduced traffic for 15 minutes |
Treat those numbers as initial controls, not universal benchmarks. A read-only inventory lookup can tolerate more failures than a refund endpoint. A booking tool with an unknown result deserves a faster stop than a search tool that returns an explicit error.
Separate the Three Breaker States
A useful circuit breaker has three states. Keep the state machine in trusted application code, not in the prompt.
Closed: normal traffic can reach the tool. The breaker records attempts, results, latency, and reason codes. “Closed” means the path is permitted, not that every call will succeed.
Open: the tool call is rejected before execution. The chatbot receives a structured reason and a permitted fallback, such as answering from static documentation, collecting details for a human, or reporting that the operation is temporarily unavailable.
Half-open: only a small number of controlled probes can run. Real customer writes should not become diagnostic traffic. Prefer a read-only health check, a synthetic account, or a reversible canary action with a known expected result.
Scope state narrowly. If the shipping-status API fails, the knowledge-base search and human handoff can remain available. A global kill switch is still valuable for emergencies, but it should not be the first response to one unhealthy dependency.
This narrower scope also keeps the customer experience honest. “I cannot check live shipping status right now” is precise. “The chatbot is unavailable” discards working paths and sends avoidable volume to support.
Map the Blast Radius Before Choosing Thresholds
The same error rate can mean very different risk. Classify each tool by the effect it can cause and the certainty of its result.
Read-only calls fetch product, order, or policy data without changing it. Their main risks are stale answers, privacy exposure, latency, and unnecessary cost. A breaker can often fall back to a sourced answer or a handoff.
Reversible writes create tickets, update preferences, or reserve a slot that an operator can undo. They need idempotency, an audit record, and a fallback that does not invite the customer to submit the same request repeatedly.
Consequential writes move money, cancel service, change access, or send an external message. One unknown result can be more dangerous than ten explicit failures. Open the write path, reconcile the destination, and require evidence before telling the customer what happened.
Security-sensitive calls fail differently again. Repeated permission denials, cross-tenant identifiers, or policy-check failures should trip an actor- or tenant-scoped breaker even when the destination itself is healthy. The tool-hook guide explains where to validate identity and authorization; the breaker decides when repeated bad outcomes mean the path must stop.
Record this classification in the tool registry. If the only copy lives in prose inside the system prompt, neither the runtime nor an auditor can enforce it reliably.
Count Outcomes, Not the Model's Intentions
The breaker needs signals from the execution layer. A fluent explanation, a tool selection, or an HTTP 200 is not a verified business result.
For every attempt, record:
- tool, operation, actor, tenant, and action ID;
- breaker state and policy version at decision time;
- start time, end time, latency, and attempt number;
- outcome class: success, definite failure, denied, unknown, or malformed;
- downstream receipt or record ID when one exists;
- model run ID and conversation ID for investigation;
- fallback selected and whether a human case was created.
Use a rolling window with a minimum sample size. A 50% error rate after two calls is noisy; 12 failures among 40 calls is a credible service problem. The calculation can stay simple:
failure_rate = definite_failures / completed_attempts unknown_rate = unknown_outcomes / submitted_writes
Do not hide unknown outcomes inside the ordinary failure count. A definite rejection is safe to retry under policy. A timeout after submission may have completed downstream and needs status lookup or reconciliation before any second write.
Trajectory monitoring complements the breaker. The multi-step failure guide helps detect loops and wrong turns across a run; the circuit breaker turns those observations into an immediate execution boundary.
Worked Case: A Ticket API Slows Down
Consider a support chatbot that can create priority tickets. The trip card opens the tool after five definite failures in 20 attempts, or after two unknown outcomes following submission.
At 09:00, the system has processed 18 ticket attempts: 16 successes, one validation failure, and one provider rejection. The breaker remains closed.
At 09:03, attempt 19 times out before the request body is sent. That is a definite transport failure. The runtime retries once with the same action ID and idempotency key.
At 09:04, the retry reaches the provider but the response times out. The destination may have created the ticket, so the runtime classifies the outcome as unknown and checks by idempotency key. The status lookup also times out.
At 09:05, another customer's submission follows the same path and becomes unknown. The breaker opens immediately. It has not waited for five ordinary failures because two writes may already exist.
The next visitor sees this exchange:
Customer: My account is locked. Please open an urgent ticket. Chatbot: The ticket system is delayed, so I cannot confirm a new ticket right now. I saved your issue for the support queue with reference AGK-4821. Please do not submit it again. A support specialist can check whether a ticket was already created and continue from there.
That answer avoids three common failures. It does not claim the ticket exists, it does not encourage a duplicate, and it gives the customer a stable reference. Behind the scenes, the queue item includes both unknown action IDs so an operator can reconcile them.
If 40 similar conversations arrive while the breaker is open, the tool receives zero additional writes. Without the breaker and with one automatic retry per timeout, those 40 conversations could create as many as 80 uncertain submissions. The breaker converts a multiplying incident into a bounded reconciliation queue.
Make the Open State Useful
An open breaker should define work, not merely return an error. Pair each trip condition with one approved degraded mode.
For a failed knowledge lookup, the bot can offer a clearly labeled general answer from a current static source. For an account-specific lookup, it should avoid guessing and collect the minimum details for handoff. For a write with an unknown outcome, it should issue a local reference, prevent resubmission, and route reconciliation to the team that owns the destination.
Preserve the evidence needed to recover, but keep sensitive data out of alerts. An alert can carry tenant ID, action ID, reason code, counts, and a link to restricted logs. It rarely needs the entire transcript or raw downstream payload.
Long work needs a durable pause as well. The async workflow guide shows how job IDs, checkpoints, and resumable states prevent a restart from repeating completed work. A breaker should pause at the action boundary and retain that state rather than asking the model to reconstruct it from conversation text.
Recover Through a Half-Open Gate
Time passing is not proof of recovery. After the cool-down period, move to half-open and send controlled probes.
Choose a probe that tests the actual failure mode. A generic health endpoint can return green while authenticated writes still fail. For a ticket API, verify authentication, create a synthetic ticket with a stable idempotency key, read it back, and remove or close it if the test environment permits.
Limit concurrency in half-open state. One probe at a time prevents a recovering service from receiving a burst of queued demand. Require several consecutive verified successes, then restore traffic gradually. If any probe produces an unknown write, reopen immediately and lengthen the cool-down.
Keep the reconciliation queue separate from recovery probes. Closing the breaker only proves that new calls are healthy. It does not establish whether earlier refunds, tickets, or bookings completed. The incident containment runbook provides the broader path for inventorying uncertain effects and restoring writes safely.
Test the Breaker as a Product Boundary
Exercise the breaker before customers encounter it. A small fault-injection suite can cover the highest-risk transitions:
| Injected condition | Expected breaker behavior | Expected customer behavior |
|---|---|---|
| Five explicit provider failures | Opens only the affected operation | Offers the documented fallback |
| Two post-submit timeouts | Opens writes and starts reconciliation | Makes no completion claim |
| Repeated permission denials | Blocks the actor or tenant scope | Reveals no protected data |
| Slow but successful responses | Trips the latency policy if the queue is at risk | Sets an honest delay expectation |
| Breaker-state store unavailable | Fails closed for consequential writes | Routes to a human path |
| First half-open probe fails | Reopens without releasing queued traffic | Keeps the degraded mode active |
| Four probes succeed | Restores a small traffic percentage | New requests proceed normally |
Also test the wording. Ask whether a customer could reasonably interpret the fallback as confirmation, retry the action from another channel, or lose the reference needed for follow-up. Operationally correct code can still create duplicate work through ambiguous language.
Assign an owner to every trip card. The integration owner sets thresholds and probes; the business owner approves degraded behavior; support owns customer wording; security owns abuse and permission signals. Review thresholds after incidents and traffic changes, not after every isolated error.
A Controlled Stop Keeps the Rest of Support Moving
More capable agents can work longer and touch more systems, which makes a bounded stop more valuable. Put the decision outside the model, scope it to the failing operation, distinguish unknown writes from explicit failures, and require verified probes before recovery. Customers then receive a truthful next step while operators get a finite incident to resolve.
In Agentkit, conversation logs can support the review trail and custom API calls can connect action workflows; enforce circuit state and side-effect guarantees in the trusted service that owns each downstream operation.
No credit card required.
