How to Crawl Large Sites with Screaming Frog
- George Apostolov
How to Crawl Large Sites with Screaming Frog: step-by-step guide
Go to Screaming Frog > Settings > System.
Select Storage Mode.
Select Database Storage mode, then click OK.
Go to Configuration > Spider > Limits.
Raise or uncheck the Max URL limit.
Go to Advanced.
Click the boxes accordingly, then click OK to apply changes.
Quick answer: To crawl large sites, configure Screaming Frog by opening Settings, System, and selecting Database Storage mode. Next, navigate to Configuration, Spider, Limits to raise or uncheck the Max URL limit. Finally, go to Advanced, adjust the boxes accordingly, and click OK to apply all changes.
About this tutorial
Optimize the app to handle massive websites by adjusting storage, URL limits, and advanced settings for a complete site crawl. This guide shows how to crawl Large Sites with Screaming Frog in 7 steps, starting with "Go to Screaming Frog > Settings > System" and finishing with "Click the boxes accordingly, then click OK to apply changes ". The interactive demo above is a click-through copy of the real product interface, so you can rehearse each step before doing it in your own account.
Frequently asked questions
How many steps does it take to crawl Large Sites with Screaming Frog?
7 steps. Each one is listed in order above, and the demo highlights exactly where to click.
Can I try it without changing my own account?
Yes. Clicking through the demo doesn't touch your data or settings. Once you know the flow, repeat it in your own workspace.
Create your first interactive demo in minutes
No credit card required • Start building today
