# How to Crawl Large Sites with Screaming Frog

> To crawl large sites, configure Screaming Frog by opening Settings, System, and selecting Database Storage mode. Next, navigate to Configuration, Spider, Limits to raise or uncheck the Max URL limit. Finally, go to Advanced, adjust the boxes accordingly, and click OK to apply all changes.

Canonical: https://livedemo.ai/tutorials/how-to-crawl-large-sites-with-screaming-frog/
Updated: 2026-10-05
Interactive demo: https://story-api.livedemo.ai/workspaces/62255d3d3304f17b8f5e2d03/stories/692a35fbab56e05a00fda81a/preview?step=1

## Steps

1. Go to Screaming Frog > Settings > System.
2. Select Storage Mode.
3. Select Database Storage mode, then click OK .
4. Go to Configuration > Spider > Limits.
5. Raise or uncheck the Max URL limit.
6. Go to Advanced.
7. Click the boxes accordingly, then click OK to apply changes .