To train a chatbot on your website, add the pages that answer customer questions, wait for processing, and test the answers against those pages before publishing. In Agentkit, you can crawl links, import a sitemap, or add individual URLs. Supplement those pages with files, text and Q&A where your website leaves a gap.
Training here means building a searchable knowledge base. It does not mean changing the underlying AI model. When someone asks a question, the agent retrieves relevant source content and uses it to compose an answer. Missing, contradictory or outdated sources can still produce a poor answer, so successful processing is only the first check.
Choose the content before crawling
Start with a clear job, such as answering setup questions for one product or explaining your returns policy. Choose current public pages that cover that job. A broad crawl can add duplicate pages and old policies that make the answer harder to verify.
| Source | Use it for | Check before adding |
|---|---|---|
| Website pages | Help articles, product pages and current policies | The URL is public and the answer is visible |
| Files | Manuals and information not published online | The file is current and extraction preserves the relevant details |
| Q&A | Specific answers missing from the other sources | The answer agrees with your current policy |
| Text | A short explanation or temporary announcement | Someone owns its review and removal |
For a file-first setup, use the document chatbot guide. Keep private account information out of a public knowledge base. An agent trained on an internal document should not automatically be published for website visitors.
Add website pages in Agentkit
Open your agent and go to Sources, then Website. Choose the input that matches the pages you want to add.
Crawl links
Use Crawl links when you want Agentkit to discover pages from a starting website URL. Review the advanced include and exclude path filters to keep the crawl focused on your chosen section.
A help-center crawl should cover the help content visitors need. Exclude login pages, tag archives, obsolete product versions and duplicate print pages. Discovery is not proof that every relevant page was successfully indexed.
Sitemap
Use Sitemap when your website already lists its intended public URLs in a sitemap. Enter the sitemap URL and use the path filters when you need only one section.
A sitemap helps discover URLs. It does not guarantee that the pages are accessible, current or useful. A sitemap can contain an old policy or a URL that now redirects, so check the results before treating the source set as complete.
Individual link
Use Individual link for a specific FAQ, pricing page or policy. This is also useful when you know the exact page you need and do not want a wider crawl.
Start with enough content to answer your chosen questions. You can expand after you know which questions remain unanswered. The website sources documentation describes source management, while the current Sources screen shows the available input modes.
Check indexed, skipped and failed pages
Wait for the background job to finish, then review the website source list. The distinction between an indexed page and a discovered URL matters.
- A skipped page can be outside your selected scope or beyond the available link quota. Review the reason before widening the crawl.
- A failed page needs investigation. Confirm it opens publicly, check that it contains useful text, and retry after fixing the cause.
- An indexed page still needs an answer check. A page can process successfully while omitting the paragraph your visitor needs.
If a page requires a login, upload an approved public version or add a suitable text source instead. For a JavaScript-heavy page, test a question whose answer appears only in that page's main content. That tells you more than assuming all visible content was extracted.
Add files, Q&A and text for the gaps
In Sources, use Files for PDF, DOCX, XLSX, PPTX, TXT or Markdown documents. The upload limit is 50MB per file and 20 files per batch. Plan storage is a separate constraint based on processed content. Check the file source documentation and your current plan before uploading a large library.
For important answers, add a Q&A source or text source with the current information. A Q&A source gives the agent relevant material; it does not guarantee an exact answer or override every contradictory document. Fix the conflicting source too.
Avoid adding another source just to increase the source count. If a pricing answer is wrong because an old PDF is still present, uploading the new price does not settle which version the agent will retrieve. Remove or correct the obsolete material and test again.
Test the answers before publishing
Open Playground and ask the questions your visitors actually ask. Write the expected answer and source beside each question. This small acceptance check separates source coverage from fluent wording.
| Illustrative question | Expected evidence | Failure to investigate |
|---|---|---|
| Can I return an opened item? | Current return rule and its exceptions | Agent cites an older policy or omits an exception |
| How do I set up the product? | Supported setup steps | Agent fills a missing step with a guess |
| Do you offer a feature not listed here? | A clear statement of what the sources establish | Agent invents an unsupported capability |
| What about the annual option? | A relevant follow-up to the preceding pricing question | Agent loses the conversation's context |
These are examples, not measured Agentkit results. Use your own questions and policies. Include questions the sources cannot answer, then check that the agent admits the limit and offers an appropriate next step.
Correct the source, retrain it, and repeat the same question. Keep the original wording so you can tell whether the change helped. When the job is support, configure a real human escalation action rather than promising a person will join without a connected destination.
Keep website answers current
Standard and Pro plans support automatic retraining of website sources. Enable it in the agent's AI training settings. The daily job reprocesses existing indexed website URLs. It does not establish that a newly published page has been discovered, an obsolete page removed, or an uploaded file replaced.
| Change | What to do |
|---|---|
| An indexed page changes | Retrain for an urgent correction, then test the changed answer |
| You publish a new page | Add or discover its URL and confirm it was indexed |
| You retire a policy or page | Review and remove the obsolete source when appropriate |
| An uploaded file changes | Remove the old version and upload the current file |
| A Q&A or text source becomes outdated | Edit or remove it and repeat the relevant test |
Automatic refresh is useful for routine website updates. It is not a substitute for reviewing contradictory sources or checking a time-sensitive correction. If extraction fails, inspect the failure before assuming the newest policy is available.
For a more detailed source review, see the source pipeline guide. It covers comparing snapshots and treating missing or failed pages as investigation items.
Publish after the answer checks pass
Open Channels and copy the installation code generated for your agent. Use the website embedding guide for your platform. Test the installed widget too, including a relevant question and the path to support when the agent cannot answer.
Create an Agentkit agent if you want to try this workflow. Start with one reader task and a manageable source set, then expand from the questions you observe.
