Documentation

Website Sources

Crawl your website to train your agent. Auto-retrain keeps content fresh.

Website sources let you train your agent by crawling pages from your website. We extract text content automatically and can keep it updated with daily auto-retraining.

Adding a Website Source

  1. Go to your agent's Sources > Website
  2. Choose Crawl links, Sitemap, or Individual link
  3. Enter the website, sitemap or page URL, such as https://yoursite.com/faq for an individual page
  4. Use include/exclude path filters for a focused crawl or sitemap import
  5. Submit and wait for processing. Review indexed, skipped and failed pages

The crawler extracts text content from the page, removing navigation, footers, and other non-content elements.

What Gets Crawled

  • Main page content and headings
  • Article text and paragraphs
  • Lists and structured content

What's excluded:

  • Navigation menus and footers
  • Images and videos (text only)
  • Login-protected pages

Adding Multiple Pages

Use a crawl or sitemap to discover multiple URLs, or add individual pages. For useful coverage, include:

  • FAQ/Help pages
  • Product documentation
  • Pricing page
  • About/Company info
  • Key blog posts

Tip: Start with 5-10 of your most important pages, test in the Playground, then add more as needed.

JavaScript-Rendered Pages

For websites with JavaScript-rendered content (React, Vue, etc.), we use the Jina Reader API for better extraction:

  • Renders JavaScript before extraction
  • Returns clean markdown
  • Handles dynamic content on modern web apps

Jina Reader runs first by default; we fall back to a traditional HTML crawl if it can't extract content.

Auto-Retraining

Standard and Pro plans include automatic daily retraining for website sources. When enabled:

  1. Your web sources are recrawled every 24 hours
  2. Content changes are automatically picked up
  3. Check processing status and test the answers affected by a change

Enable Auto-Retrain

  1. Open your agent's AI settings
  2. Find the training settings
  3. Enable Auto-retrain daily and save

The daily job refreshes existing indexed website URLs. Add new pages separately, review obsolete sources, and replace uploaded files yourself. Test an urgent correction after retraining rather than waiting for the next scheduled run.

PlanAuto-Retrain
FreeNot available
HobbyNot available
StandardDaily
ProDaily

Manual Retraining

For immediate updates or to retrain specific pages:

  1. Go to Sources > Website
  2. Find the source you want to refresh
  3. Click the Retrain button
  4. Wait for processing to complete
PlanWebsite Links
Free10
HobbyUnlimited
StandardUnlimited
ProUnlimited

Troubleshooting

"Page couldn't be crawled"

  • Verify the URL is publicly accessible
  • Check if the page requires login
  • Try a different page from your site

"No content extracted"

  • The page may be JavaScript-heavy (Jina Reader handles most JS pages)
  • Check if content is behind tabs or accordions
  • Ensure the page has actual text content

"Content seems outdated"

  • Enable auto-retraining for automatic updates
  • Manually retrain the source
  • Check the "Last trained" timestamp

Next Steps