In June 2026, Inception's Mercury 2 crossed 1,000 tokens per second. It is a diffusion language model — instead of generating one token at a time like every mainstream LLM since GPT-2, it generates tokens in parallel and refines them through iterative denoising, hitting roughly 10x the throughput of the fastest autoregressive models. The tradeoff is real but narrow: 5 to 15 percent lower quality on hard reasoning, near-parity on structured output, translation, and classification. Around the same time, Google shipped Gemini 3.5 Flash, the fastest model scoring above 70 on aggregate quality benchmarks at about 284 tokens per second, beating the previous-generation Gemini 3.1 Pro on coding and agentic tasks while running several times faster.
The headlines went to the speed numbers. The more useful story for anyone running a chatbot is what those numbers do to a decision most teams get backwards: which model to put behind the widget.
The instinct is to pick the smartest model available. For a chatbot, that instinct is usually wrong. Most of what a support or sales chatbot does is not hard. It is "what are your hours," "do you ship to Canada," "how do I reset my password," "what's the difference between your plans." On questions like these, a fast model and a flagship produce answers a customer cannot tell apart — except the fast model produces it in a second and the flagship makes them wait four. In chat, that wait is not a minor annoyance. It is the single biggest predictor of whether the conversation continues.
Why latency decides the conversation
A chatbot lives in a different latency budget than a search box or a report generator. The interaction is a back-and-forth, and the user is sitting there watching a typing indicator. Every turn that takes too long invites them to give up and either leave or escalate to a human — which defeats the point of the chatbot.
Three numbers determine how fast a reply feels, and they are not the same thing:
| Metric | What it measures | Why it matters in chat |
|---|---|---|
| Time to first token (TTFT) | How long until the first word appears | Dominates perceived responsiveness — silence before the first token feels like a hang |
| Tokens per second | How fast text streams once it starts | Determines how long a full answer takes to finish rendering |
| Total response time | TTFT plus generation time for the whole answer | The real number, but users judge by the first two |
The trap is optimizing only for the benchmark number, which is usually tokens per second. A model that streams at 800 tokens per second but takes two seconds to produce its first token feels slower than a model at 200 tokens per second that starts in 300 milliseconds. For a short chatbot answer — most are under 80 words — time to first token and total length matter more than raw streaming speed, because the answer finishes before peak throughput ever comes into play.
This is why "pick the smartest model" backfires. Flagship reasoning models often think before they answer, adding latency before the first token even appears. On a hard escalation that is worth the wait. On "what are your hours," the customer has already opened a second tab.
The speed frontier in 2026
For most of the LLM era, faster meant dumber, full stop. That correlation is breaking. Two things changed in 2026.
Fast tiers got genuinely good. The small-and-fast models — Gemini 3 Flash, Claude Haiku 4.5, GPT-5.4 mini and nano — now answer the bulk of routine chatbot traffic at a quality a customer cannot distinguish from a flagship. The gap between tiers collapsed on easy tasks even as it widened on hard ones.
A new architecture attacked latency directly. Diffusion LLMs like Mercury 2 generate in parallel rather than left-to-right, which is a structural speedup rather than an incremental one. They are not the right pick for multi-step reasoning yet, but for the classification, extraction, and short-answer work that makes up most of a chatbot's job, they point at where the floor is heading.
Here is the rough shape of the tradeoff as it stands today. Exact numbers move weekly; the pattern is what matters:
| Model class | Relative speed | Quality on routine chat | Best chatbot use |
|---|---|---|---|
| Diffusion (e.g. Mercury 2) | Fastest (~10x) | Strong on short answers, classification, extraction | Intent routing, FAQ, structured replies |
| Fast tier (Gemini Flash, Haiku 4.5, GPT mini/nano) | Very fast | Near-flagship on common questions | Default for most support and sales traffic |
| Flagship (GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro) | Slowest | Best on multi-step reasoning | Escalations, policy edge cases, complex troubleshooting |
The fast tier is the right default for a chatbot, not the compromise. You move up to a flagship for the specific queries that need it, not for the whole workload.
When smarts actually wins
Speed is the right default, not a universal rule. There is a real set of chatbot moments where a slower, smarter model earns its latency:
- Multi-constraint policy questions. "I'm past the 30-day return window but the item arrived damaged and I'm a Pro customer" requires holding several conditions at once. A fast model flattens nuance here; a flagship reasons through it.
- Troubleshooting with branching logic. Diagnosing why an integration is failing involves hypotheses, follow-ups, and revisiting earlier answers. Depth beats speed.
- High-stakes or regulated answers. When a wrong answer carries legal or financial consequences, the extra seconds are cheap insurance. Our guide to chatbot hallucination and liability covers where that line sits.
The customer tolerates waiting in exactly these cases, because the question was obviously hard. The mismatch that hurts you is the opposite one: making someone wait four seconds to hear your store hours.
Make answers feel faster without changing the model
Model choice is the biggest lever, but it is not the only one. Perceived speed is partly a UX problem, and a few changes help regardless of which model you run.
Stream the response. Showing text as it generates collapses the wait. A streamed answer at 200 tokens per second feels faster than a non-streamed answer at 800, because the user sees motion immediately instead of staring at a blank bubble.
Keep answers short. A 60-word answer finishes before a 200-word answer is half done, on any model. Most chatbot questions want a direct answer, not an essay. Instructing the model to be concise is one of the highest-return prompt changes you can make — it cuts latency and reduces the chance the model wanders into a wrong tangent. See chatbot prompt engineering for prompt patterns that keep responses tight.
Cut the context you do not need. A retrieval-augmented answer that stuffs 30 chunks into the prompt is slower on every model than one that retrieves the 4 chunks that matter. Retrieval quality is a latency feature, not just an accuracy one. The chatbot optimization guide breaks down how context size affects both.
Route trivial turns away from the model entirely. Greetings, "thanks," and obvious menu picks do not need an LLM call at all. A canned response or a suggested-message button is instant and free.
Picking a fast default in practice
This is a settings decision, not an engineering project. In Agentkit, the model is a per-chatbot dropdown spanning 8 models across OpenAI, Anthropic, and Google — including the fast tier that should be most chatbots' default: GPT-5.6 Luna, Claude Haiku 4.5, and Gemini 3.7 Flash. New chatbots default to GPT-5.6 Luna precisely because the fast tier handles the overwhelming majority of real traffic, and defaulting to a flagship would make every customer wait for reasoning that most questions never need.
You can switch models from the settings page with no re-training and no code change, which makes the right workflow simple: start on a fast model, watch your real conversations, and only move specific high-difficulty traffic up a tier. If you run more than one chatbot — Standard supports 2, Pro supports 3 — a common split is a fast Haiku- or Flash-backed bot on the public site for high-volume FAQ, and a flagship-backed bot inside the product for authenticated support where questions get harder. That is the latency-first version of the multi-model strategy, which approaches the same model split from the cost angle.
Whatever you pick, measure it. Track time to first token and full response time alongside resolution rate, not in isolation — a model that resolves 2 percent more conversations but adds three seconds to every reply may lose you more customers to abandonment than it saves. The chatbot KPIs guide covers which metrics actually correlate with retention.
Common mistakes
The failure modes here are consistent across the teams that get this wrong:
Defaulting to the flagship "to be safe." This is the most expensive mistake, and it costs you in abandonment, not just inference. The safe default for a chatbot is fast; safety on hard queries comes from routing up, not from starting high.
Benchmarking on tokens per second alone. The number that ships in marketing is throughput. The number your customer feels is time to first token plus answer length. Optimize the experience, not the spec sheet.
Letting answers sprawl. A smarter model with no length instruction often produces longer answers, which means it is slower and more likely to bury the point. Brevity compounds with model speed.
Never revisiting the choice. The fast tier in mid-2026 is not the fast tier from six months ago. Models that were a clear quality compromise last quarter may now clear your bar. Re-test when a new fast model ships. The broader GPT vs Claude vs Gemini comparison is worth rereading each cycle.
The bottom line
The race the labs are running is a quality race, and it is genuinely impressive — Mercury 2's parallel generation and Gemini 3.5 Flash's near-flagship quality at a fraction of the latency are both real advances. But the model you put behind a chatbot is not chosen by leaderboard rank. It is chosen by the shape of your traffic, and most chatbot traffic is routine questions where a one-second answer beats a four-second one that is no more correct.
Start fast. Keep answers short. Stream them. Move only the hard queries up to a smarter, slower model, and only after you have watched real conversations tell you which queries those are. Customers will forgive a chatbot that is occasionally less clever. They will not forgive one that makes them wait — they will just leave, and you will never see the conversation that did not happen. If your chatbot is running on a flagship by default, the fastest improvement you can ship this week is switching the dropdown.
No credit card required.
