The model behind a chatbot affects how it follows instructions, uses retrieved content, and phrases an answer. It does not fix weak sources or vague instructions. The useful question is not which provider wins in general. It is which current model gives the best answers for your content, visitors, and response rules.
Agentkit currently offers twelve models from OpenAI, Anthropic, and Google. This guide uses that live catalog and avoids comparisons with retired models that no longer appear in the model picker.
The current model catalog
| Provider | Models available in Agentkit |
|---|---|
| OpenAI | GPT-6 Astra, GPT-6 Sol, GPT-6 Luna, GPT-5.6 Sol, GPT-5.6 Terra |
| Anthropic | Claude Fable 5.1, Claude Opus 5.5, Claude Opus 5, Claude Sonnet 5, Claude Haiku 4.5 |
| Gemini 3.7 Flash, Gemini 3.1 Pro |
New agents on every plan, including Free, start with GPT-6 Sol. In our evaluation, it was the most reliable at answering from your content and at running actions and multi-step tools. Existing agents keep their selected model.
What to compare
Public model benchmarks rarely match a website chatbot's real job. A useful evaluation measures the behavior your visitors will notice.
| Criterion | What it means | How to test it |
|---|---|---|
| Grounding | The answer stays within your sources | Ask company-specific questions, including questions your sources do not answer |
| Instruction following | The model respects your tone, scope, and escalation rules | Try direct requests, follow-ups, and attempts to push it off topic |
| Completeness | The answer includes the necessary detail without burying it | Test policies, comparisons, and multi-part questions |
| Conversation quality | Follow-up answers use the earlier exchange correctly | Run the same five-to-ten-message conversation with each model |
| Language quality | The response is natural in the languages you support | Use the same customer questions in each target language |
| Response time | The chat feels responsive on your site | Measure real widget conversations, not an isolated provider demo |
Retrieval settings, source quality, and the instructions remain part of every result. Keep them identical while comparing models.
GPT-5.6 models
Agentkit's GPT-5.6 choices are GPT-5.6 Sol and GPT-5.6 Terra. Compare these two models using your evaluation set to see whether switching improves the answers enough to matter.
Start with GPT-6 Sol because it is the current default. Add GPT-5.6 Sol when your test set contains the hardest policy, troubleshooting, or comparison questions. Test GPT-6 Luna for straightforward, high-frequency exchanges. These are testing roles, not capability guarantees.
Claude models
Anthropic choices include Claude Opus 5, Claude Sonnet 5, and Claude Haiku 4.5. Run the same instructions and sources through each model. Pay close attention to refusals, unsupported details, answer length, and whether the tone remains consistent late in a conversation.
Opus 5 and Sonnet 5 belong in a shortlist when answer quality on complex examples matters most. Haiku 4.5 gives you a third Anthropic comparison point for simpler exchanges. The winning choice still depends on your own transcripts.
Gemini models
Agentkit offers Gemini 3.7 Flash and Gemini 3.1 Pro. The catalog previously contained older Flash references, so check the picker rather than copying a model name from an old setup guide.
Compare both Gemini models on long source passages, multilingual questions, and follow-up conversations if those tasks matter to your chatbot. Do not assume that the word "Pro" or "Flash" settles the result. Measure the finished response in Agentkit with your retrieval pipeline active.
Build a fair model test
The fastest reliable comparison uses a small, fixed evaluation set.
- Collect 20 to 30 real customer questions. Include easy questions, ambiguous wording, missing-answer cases, and questions that should trigger an action or escalation.
- Use the same sources, instructions, and action settings for every model.
- Start with GPT-6 Sol, then choose at least one model from each other provider family.
- Score each answer for correctness, grounding, completeness, tone, and response time.
- Reject any answer that invents a product detail, even if the rest sounds polished.
- Repeat the test after a major source or instruction change. The best model can change when the task changes.
A five-point score is enough if the rubric is concrete. Keep the original answer beside the score so reviewers can see why one model won.
A practical shortlist
If you do not yet have enough traffic to build a large evaluation set, use this sequence:
- Keep the default GPT-6 Sol as the baseline.
- Compare Claude Sonnet 5 and Gemini 3.7 Flash on the same questions.
- Add GPT-5.6 Sol, Claude Opus 5, or Gemini 3.1 Pro for the difficult examples that separate the first three.
- Include GPT-6 Luna or Claude Haiku 4.5 if your workload has many short, routine questions.
This produces evidence about your chatbot instead of a generic provider ranking.
Cost and plan considerations
Agentkit plans use message allowances rather than exposing a separate per-token bill for each model. Paid plans include ten core models, Pro adds GPT-6 Astra and Claude Fable 5.1, and the free plan uses a smaller selection. Check the pricing guide for the current plan limits.
Because the model is a chatbot setting, you can compare models while keeping the same website sources, documents, Q&A pairs, and text snippets. A model switch applies to future conversations and does not require you to ingest those sources again.
The bottom line
There is no permanent GPT-versus-Claude-versus-Gemini winner for every website chatbot. Agentkit's current picker gives you twelve specific choices. Start with GPT-6 Sol, test a cross-provider shortlist against real questions, and keep the model that makes the fewest consequential mistakes.
Review conversations after launch. When the catalog, your content, or your customer questions change, rerun the same evaluation set instead of relying on an old recommendation.
No credit card required.
