> Crawl your marketing site, docs or help center into the AI Demo Agent's knowledge and keep it up to date automatically.

Canonical: https://livedemo.ai/docs/ai-demo-agent/crawl-your-website/

# Crawl your website

The fastest way to teach the agent: point it at your site.

## Add your site

1. Open the agent's **Knowledge** tab and click **Add source**.
2. Choose **Whole website (crawl)**.
3. Enter your site's URL.
4. Set **Max pages** (1–100, default 25) and **Re-read every** N hours (default 24, 0 = never).
5. Add it. The source shows as ready when crawling is done.

## How it reads your site

- It uses your `sitemap.xml`. Without one, it follows links on the same domain.
- Pages are rendered in a real browser, so JavaScript sites work.
- Menus, footers, cookie banners and pop-ups are skipped. Headings, lists, tables and image alt text are kept.
- Re-crawls skip unchanged pages.

## Which sites to add

| Site | Why |
| --- | --- |
| Marketing site | Features, pricing, use cases |
| Docs or help center | Detailed how-to answers |
| Changelog | What's new |

Add each as its own source, such as `https://acme.com` and `https://docs.acme.com`, so each gets its own page budget.

## Check the result

Click 👁 on the source to see every page and the exact text the agent read. Pages with almost no text get a warning: usually a login wall or content drawn as images. See [Knowledge](https://livedemo.ai/docs/ai-demo-agent/knowledge/#see-what-the-agent-reads).

## Only need one page?

Use **Single page URL**, such as your pricing page. It also re-reads on a schedule.

---

Next: [Knowledge gaps](https://livedemo.ai/docs/ai-demo-agent/knowledge-gaps/)