OpenAI's July guidance on managing AI investments recommends measuring useful work per dollar and matching capacity to proven demand. That advice matters well below enterprise scale. A website chatbot can exhaust a message allowance, slow down during a launch, or create a review backlog long before its monthly token bill looks alarming.
Start with one row per demand scenario and six inputs. This compact capacity card is the asset to keep beside every forecast:
| Input | Formula or source | Why it matters |
|---|---|---|
| Conversations | Website visits × chat-start rate | Converts traffic into support demand |
| Model calls | Conversations × AI turns × retry factor | Captures multi-turn chats and technical retries |
| Peak concurrency | Conversations in peak hour × active minutes ÷ 60 | Exposes latency and queue pressure hidden by monthly totals |
| Variable cost | Model + retrieval + tool + observability cost | Shows the spend that grows with usage |
| Review hours | Conversations × review rate × minutes per review ÷ 60 | Sizes the human quality workload |
| Durable resolutions | Conversations × resolution rate × (1 − reopen rate) | Provides the denominator for cost per useful outcome |
Build a baseline from observed data, then calculate a launch case, an incident case, and a failure case. The spread between those rows is more useful than a single precise-looking estimate.
Forecast Conversations Before Tokens
Capacity planning begins with customer behavior. Token estimates built directly from page views skip the decision that creates demand: whether a visitor opens the chatbot and continues beyond the first message.
Use at least four weeks of traffic and conversation data when available. Split it by weekday, hour, referral source, page type, and major customer segment. A pricing page may start fewer chats than a help article but produce longer, more expensive conversations. A product launch may increase both the number of visitors and the percentage who need help.
For each segment, record:
- visits or eligible users;
- chat-start rate;
- average AI turns per conversation;
- p90 turns, so a small long tail stays visible;
- average active conversation time;
- resolution and reopen rates;
- tool-call, retrieval, and handoff frequency.
Do not treat every message as one model call. A single customer turn may trigger retrieval, a model response, a validation retry, and a tool-result explanation. Conversely, an opening message or deterministic button click may require no model call at all. Trace the actual path and calculate a retry factor from observed requests.
When no history exists, use an explicit assumption range. For example, forecast chat starts at 3%, 6%, and 10% of visits. Label those values as assumptions and replace them after the first week. False precision is especially dangerous when a new widget placement changes visitor behavior.
Worked Capacity Sheet for a Support Chatbot
Assume a support site expects 24,000 visits next month. The baseline chat-start rate is 6%, so the forecast contains 1,440 conversations. Each conversation averages 5.4 AI turns, and logs show an 8% overhead from validation retries and recovered provider requests.
The model-call estimate is:
1,440 conversations × 5.4 AI turns × 1.08 retry factor = 8,398 model calls
Suppose each call averages 1,800 input tokens and 220 output tokens. That produces about 15.12 million input tokens and 1.85 million output tokens. For this worked example, use illustrative rates of $0.60 per million input tokens and $2.40 per million output tokens. Substitute the contracted rates and cache behavior from your own provider.
| Monthly component | Calculation | Forecast |
|---|---|---|
| Input tokens | 15.12M × $0.60/M | $9.07 |
| Output tokens | 1.85M × $2.40/M | $4.44 |
| Retrieval and tool usage | 1,440 × $0.004 | $5.76 |
| Observability | 1,440 × $0.0015 | $2.16 |
| Human review sample | 1,440 × 5% × 6 min × $30/hour | $216.00 |
| Total measured operating cost | Sum of the rows | $237.43 |
Now calculate the useful denominator. If 78% of conversations appear resolved and 8% of those reopen within 72 hours, the chatbot produced about 1,033 durable resolutions:
1,440 × 0.78 × (1 − 0.08) = 1,033.3 durable resolutions $237.43 ÷ 1,033.3 = $0.23 per durable resolution
The example exposes a common planning error. Model inference costs $13.51, while the planned review sample costs $216. Cutting output tokens by 20% saves less than one dollar here. Reducing unnecessary review time, improving the sampling strategy, or preventing reopened conversations has much more impact.
Keep labor visible even when the same support team absorbs it. Unpriced review work still consumes capacity. If the forecast asks a two-person team to review 300 conversations during a launch week, the operational limit arrives before the API budget.
Convert Monthly Demand Into Peak Load
A monthly average cannot predict a slow chatbot at 10:05 on launch morning. Estimate peak concurrency separately.
Suppose 120 conversations begin during the busiest hour and remain active for an average of eight minutes:
120 conversations × 8 active minutes ÷ 60 = 16 concurrent conversations
Concurrency is only the first layer. If users exchange messages in bursts, requests may cluster within the same minute. Record peak model requests per minute, p95 response time, tool latency, provider errors, and queue wait. Load-test above the forecast rather than exactly at it; a 25% to 50% margin is a reasonable starting assumption until production evidence supports a narrower one.
Model choice changes the shape of the load. Higher reasoning effort may increase output, tool use, and response time even when conversation volume stays flat. Route routine intents to the least expensive configuration that passes the quality bar, then reserve longer work for cases that benefit from it. The reasoning-effort framework shows how to set those boundaries by intent.
Track downstream capacity too. A chatbot that creates support tickets faster than people can accept them has moved the queue rather than removed it. Include handoff arrivals per hour, time to qualified owner, and the receiving team's safe queue depth.
Give Every Scenario a Trigger and Response
Build at least four forecast rows. Each row should change both demand and behavior assumptions.
| Scenario | Assumption changes | Trigger to watch | Prepared response |
|---|---|---|---|
| Baseline | Recent median traffic and chat behavior | Forecast error exceeds 15% for two weeks | Refit the baseline |
| Launch | Higher visits, chat-start rate, and setup questions | Requests/minute or p95 latency crosses the launch limit | Narrow low-value intents, add staffed coverage |
| Incident | Concentrated status questions, repeat contacts, handoffs | Canonical incident answer is missing or queues exceed safe depth | Publish approved incident content, pause risky actions |
| Provider degradation | More timeouts, retries, and fallback calls | Error or retry factor passes the agreed threshold | Switch to tested fallback or reduce scope |
| Abuse | Abnormal starts, message depth, or repeated prompts | Per-session or per-domain budget is exceeded | Apply rate limits and preserve legitimate access |
The AI support surge plan covers the operational response to launches and incidents. Capacity planning supplies the numbers that decide when that plan begins.
Attach one owner to each trigger. “Watch latency” leaves the decision open during the worst moment. “The on-call support lead disables nonessential tool actions when p95 response time exceeds 12 seconds for 10 minutes” is executable.
Put Guardrails Around Spend and Service
A useful budget has several boundaries:
Per conversation. Cap turns, elapsed time, tool calls, retrieved documents, and high-effort retries. When a limit is reached, answer narrowly, ask one clarifying question, or hand off according to the intent.
Per visitor or account. Prevent one session, customer, or embedded domain from consuming the shared allowance. Preserve a reasonable path for legitimate complex cases.
Per hour. Protect response time and downstream tools during bursts. Queueing may be safer than sending unlimited concurrent requests into a dependency that is already failing.
Per day and month. Alert before a plan allowance or financial ceiling becomes an outage. Use at least a warning threshold and an action threshold, each with a named response.
Per outcome. Review cost per durable resolution, qualified lead, completed booking, or other useful result. A flat token bill can hide worsening outcomes when reopen rates or abandonment climb.
The chatbot rate-limit guide goes deeper on session, account, and domain controls. Keep those protective limits separate from the forecast: the forecast predicts likely demand, while the guardrail defines what the system will permit.
Never let a cost ceiling silently turn into confident but lower-quality answers. If the approved model, source, or tool is unavailable, the chatbot should disclose the limitation and offer the documented fallback. Cost controls must preserve the boundary between “answered” and “could not verify.”
Reforecast From Outcomes Every Week
Compare forecast and actual values at the same grain. A monthly total may be accurate while hiding a launch-hour miss that produced abandonment. Review conversations, model calls, peak requests, p95 latency, retries, handoffs, review hours, durable resolutions, and cost per durable resolution.
Investigate variance rather than automatically adding capacity. More calls per conversation may come from complex demand, a prompt that became verbose, retrieval failures, repeated tool errors, or customers asking again because the first answer was weak. Each cause needs a different fix.
Use the chatbot KPI definitions to keep resolution, fallback, handoff, and response-time terms consistent. Then preserve the assumptions behind every forecast: date range, traffic source, model configuration, source version, retry rule, labor rate, and reopen window. Without that record, next month's comparison becomes guesswork.
Capacity Should Follow a Customer Promise
Chatbot capacity planning connects traffic, model work, human review, and durable outcomes in one forecast. Begin with observed conversations, expose the assumptions, test peak load, and predefine the response to each threshold. The result is a service plan the support team can operate, not merely a token estimate finance can approve.
In Agentkit, conversation logs and analytics can supply the observed volume and outcomes for the weekly reforecast.
No credit card required.
