How to Crawl Large Sites with Screaming Frog
- George Apostolov
- 05 Sep 2025
Guide
Go to Screaming Frog > Settings > System.
Select Storage Mode.
Select Database Storage mode, then click OK.
Go to Configuration > Spider > Limits.
Raise or uncheck the Max URL limit.
Go to Advanced.
Click the boxes accordingly, then click OK to apply changes.
Create your first interactive demo in minutes
No credit card required • Start building today