OpenAI reported on September 6 that its research organization now uses 3.1 agent-workdays for every human workday. Four or more concurrent agent sessions are becoming more common, and agents still need substantial human steering as tasks get harder. That is a useful warning for customer-facing automation. Agent capacity can rise much faster than human attention.
Start with a risk-weighted supervision budget. These are conservative launch limits, not universal benchmarks:
| Work class | Customer effect | Starting live limit per operator | Oversight rule |
|---|---|---|---|
| Read-only | Search public help content | No fixed live limit | Sample completed conversations |
| Reversible | Create a draft or support ticket | 8 active runs | Alert on uncertainty and repeated failure |
| Consequential | Change a booking, account, or subscription | 3 active runs | Review before commit and on any unknown result |
| Irreversible or privileged | Refund money, delete data, change access | 1 active run | One named approver, no unattended commit |
Treat those limits as hypotheses. Measure the attention each class consumes, test peak conditions, and lower concurrency before operators start missing decisions.
A supervision ratio is an attention budget
"One person can supervise 20 agents" sounds precise but says almost nothing. Twenty agents answering public questions create a different workload from two agents changing customer accounts. The useful denominator is not the number of agents. It is the number of interventions they can demand during the same minute.
Calculate expected oversight demand for each work class:
oversight demand per hour = active runs x interventions per run per hour x minutes per intervention
Then compare total demand with staffed capacity:
staffed capacity per hour = operators x 60 minutes x target occupancy
Use a target occupancy below 100%. Seventy percent is a sensible starting point for a new workflow because operators need room for correlated failures, handoffs, and an incident that does not arrive on schedule. A queue that works only when every person spends every minute clearing alerts is already overloaded.
Expected demand is only the first check. Also model a burst where one dependency fails and many runs ask for help together. Average load can look comfortable while a ten-minute spike leaves consequential actions waiting without an owner.
Classify work before setting concurrency
Assign the risk class from the possible customer effect, not from how easy the prompt looks.
A shipping-policy answer is read-only even if retrieval takes several steps. Creating a support ticket is usually reversible because a person can close or correct it. Moving an appointment may be consequential because another customer can take the released slot. Issuing a refund or deleting an account deserves the strictest controls even when the API call takes one second.
Use the highest-risk step to classify a mixed workflow. An agent that searches documentation, checks an order, and then cancels it belongs in the cancellation class. Do not average a dangerous final action down because the earlier steps were harmless.
Define the class in the action contract:
- customer state that can change;
- maximum value or scope;
- whether the effect can be reversed cleanly;
- approval required before commit;
- evidence required after commit;
- timeout and unknown-result behavior;
- queue and owner when automation stops.
The chatbot approval workflow explains how to bind a person's confirmation to exact parameters. Supervision adds a capacity question: will a qualified person actually be available when several approvals arrive together?
Worked example: two operators and eighteen active runs
Suppose a support team staffs two operators for chatbot oversight. Each operator can spend 70% of the hour on live monitoring, leaving the rest for customer replies, coordination, and recovery work.
Capacity = 2 x 60 x 0.70 = 84 operator-minutes per hour
The chatbot currently has these active runs:
| Work | Active runs | Intervention rate | Minutes each | Hourly demand |
|---|---|---|---|---|
| Drafting support tickets | 12 | 0.25 per run | 4 | 12 minutes |
| Changing subscriptions | 5 | 0.8 per run | 6 | 24 minutes |
| Processing one approved refund | 1 | 2 per run | 10 | 20 minutes |
| Scheduled pre-commit approvals | 6 per hour | 1 each | 3 | 18 minutes |
Total expected demand is 74 operator-minutes per hour. The team has ten minutes of headroom. That may pass the average-load check, but it fails the absence test. If one operator steps away, capacity falls to 42 minutes while demand stays at 74.
The fix is mechanical. Pause new subscription changes and refunds when only one operator is available. Let public answers continue, queue ticket drafts, and route urgent customers to a normal support path. The system should reduce its permissions before the remaining operator has to choose which alert to ignore.
Run the same calculation with observed intervention rates after launch. If subscription changes generate 1.4 interventions per run instead of 0.8, the class consumes 42 minutes per hour, not 24. Lower the live limit or repair the cause before expanding traffic.
Separate continuous monitoring from review
Not every conversation needs a person watching it live. Forcing live review onto low-risk answers burns attention and trains operators to click through routine alerts. Use three oversight modes.
Automatic policy checks run on every conversation. They enforce permissions, validate parameters, detect repeated failures, and stop actions outside scope. These checks should block the tool call before it reaches the destination.
Live human supervision covers consequential work, unknown results, policy conflicts, and incidents. The operator receives a decision, supporting evidence, and a deadline. They should not have to reconstruct the full conversation while a commit timer runs.
Post-run sampling measures answer quality and finds quiet failures in completed low-risk work. Keep the random baseline separate from targeted risk review. The chatbot QA sampling guide provides a practical two-queue design.
Anthropic described this split after recent agent security incidents. Its August 31 security update says real-time classifiers now block suspicious tool calls, end the task, and alert a human. It also expanded offline monitoring across internal agent use. The lesson applies well outside cyber evaluations: automatic controls protect the boundary, live supervision handles urgent judgment, and retrospective review finds patterns neither caught alone.
Make every alert answerable
An alert that says "agent needs help" transfers the agent's ambiguity to the operator. Put the decision at the top and include only the evidence needed to make it.
A consequential-action alert should show:
- requested action and normalized parameters;
- authenticated customer and account scope;
- risk class and reason for review;
- current state, including whether anything already changed;
- source or system evidence used;
- safe choices such as approve, reject, ask customer, or escalate;
- deadline and automatic outcome if nobody responds;
- stable action ID for later reconciliation.
Choose a safe timeout. A missed approval should expire or move to a staffed queue. It must never become approval by silence. If the destination may already have committed an action, the timeout state is unknown, and the system should check the authoritative record before anyone retries.
The action verification guide shows how to distinguish accepted, confirmed, pending, and failed outcomes. Reuse those states in the supervision queue so the operator can see whether they are deciding a future action or recovering an earlier one.
Put concurrency limits where the action happens
A dashboard warning does not stop the seventh refund from starting. Enforce limits in the workflow runtime and in the service that owns the customer change.
Use separate limits by tenant, action type, and risk class. One noisy customer should not fill every approval slot. A slow booking provider should not pause read-only support. Keep a stricter global ceiling as protection against a wider failure.
For consequential actions, acquire an oversight slot before execution. Give the slot an expiry, bind it to the action ID, and release it only when the action reaches a confirmed, failed, or handed-off state. If the worker crashes, a reconciliation process must inspect the system of record before making the slot available again.
Queue visibility matters. Operators need the oldest item, highest risk, customer wait time, and remaining deadline. Product teams need rejected, expired, and abandoned counts. A queue can stay technically healthy while customers wait long enough to submit the same request through another channel.
Define degraded mode before attention runs out
Write the capacity stop condition into the operating procedure. Good triggers include operator occupancy above the target for 15 minutes, a high-risk item with no owner, approval latency above the customer promise, a surge of unknown action results, or loss of the primary operator.
For each trigger, specify what continues:
- public information answers may remain available;
- reversible drafts may queue with an honest wait message;
- consequential commits pause before execution;
- irreversible actions move to a staffed manual path;
- pending unknown results reconcile before any retry.
The existing AI support SOP template is the right place to record the owner, reduced-service behavior, and restoration proof. Test it during a normal shift. Waiting for an incident to discover that the pause control needs administrator access wastes the exact attention the control was meant to protect.
Measure missed attention, not agent busyness
Agent runtime and message volume tell you how much automation ran. They do not show whether supervision kept up. Track metrics that expose attention failures:
- operator occupancy by risk class;
- approval wait time at p50 and p95;
- high-risk minutes without a named owner;
- alerts that expired before review;
- actions paused because capacity was unavailable;
- unknown results waiting for reconciliation;
- customer retries while an action was pending;
- interventions that prevented a wrong action.
Review false alarms too. If operators override the same safe alert every day, refine the rule or the workflow. Do not quietly raise the concurrency limit to hide alert fatigue.
Run a saturation drill before each major expansion. Double the arrival rate, remove one operator, delay a downstream API, and inject several ambiguous results. Verify that the runtime stops high-risk work, preserves low-risk service, and gives every pending action one durable owner. The agent incident-response runbook covers the containment and recovery path when that drill exposes a gap.
Scale authority only after attention scales
More capable agents make it cheap to start work. They do not make human judgment infinitely divisible. Set concurrency from measured intervention demand, reserve room for bursts, and reduce permissions automatically when staffed capacity falls.
If your chatbot uses custom API calls or webhooks, enforce the action limits and approval state in the system that owns the change. In Agentkit, conversation logs and analytics can supply the completed conversations for sampled review.
No credit card required.
