Features / Crawl controls
Crawl settings you control, for every website
SEOSage has no hidden limits. Set the pages, speed, JavaScript and sign-in rules for each site, and crawl staging sites before they go live.
Jump to: What to crawl · How to crawl · Staging sites · Questions
See crawl settings in 25 seconds
Set what to crawl and how, for each site: limits, subdomains, URLs to skip, extra start URLs, JavaScript, robots.txt, speed, logins and schedules.
Read the video transcript
Crawl the way you want. Set the limits. Pages, depth, subdomains, and URLs to skip. Choose how it crawls. The bot, JavaScript, robots.txt, and speed. A staging site behind a password? Add a login, even the login page address. Or schedule a crawl. Start your fourteen-day free trial at seosage.co.
Decide what gets crawled
Every website is different. Set the rules for each site in its project settings, before you start.
- No page limit: Max pages and Max depth both start at 0, which means no limit
- Crawl subdomains, such as blog. or shop.
- Exclude URLs matching a pattern, for example /account/* or *?utm_*, or include only URLs matching one
- Remove URL parameters, so ?ref= variants of a page count as one URL
- Additional start URLs, plus the Sitemap URL and robots.txt URL
Scope
*?utm_*
Sources
Auto-filled after a crawl when found on the site. Every sitemap URL is always crawled.
Additional start URLs
Authentication
For crawling password-protected or staging sites. Stored locally on this device.
Custom headers
Schedule
Decide how it crawls
Choose how SEOSage behaves while it crawls, and how fast.
- Crawl as the SEOSage desktop bot, or another user agent
- Render JavaScript, to see content that scripts add to the page. It makes a crawl about four times slower, so leave it off unless you need it. If a page is only an empty app shell, SEOSage flags it as Content needs JavaScript to appear and tells you to turn this on New
- Respect robots.txt, or turn it off
- Follow nofollow links, and check the status of external links
- Set the crawl speed in URLs per second
Crawl
Limits
Crawl sites that are not public
Because SEOSage runs on your own computer, you can audit a website before it goes live.
Password-protected and staging sites
- Add a username and password. They are stored on your computer only
- For a site with a sign-in form, paste the address of the login page and SEOSage signs in with your username and password before it crawls. If the sign-in fails, you are told
- Send custom headers with every request, one per line
- A local or staging address you start on purpose is crawled as normal. A website that tries to send the crawler into your own network, such as your router, is refused and shown as a notice New
Repeat crawls
- Schedule a site to be crawled daily, weekly or monthly
- Scheduled crawls run while SEOSage is open or running in the menu bar
- Pause a crawl and resume it later, or stop it any time and keep the results so far New
- A crawl saves itself every two minutes. If the app closes or crashes, Resume crawl carries on from the saved pages instead of starting again New
See how the results look on the SEO crawler page, or test a robots.txt rule with the free robots.txt tester.
What are SEO crawl controls?
SEO crawl controls are the settings that determine how a website crawler behaves when auditing your site. They let you define which pages to include or exclude, how fast the crawler should move, whether it should render JavaScript, how it should handle authentication on private sites, and whether it should respect or override your robots.txt rules. Good crawl controls are important because a site audit is only accurate if the crawl itself matches what Google actually sees.
Without crawl controls, a crawler either crawls everything it finds (including URL parameters, session IDs, or pagination variants that create duplicate content issues) or misses pages that are behind a login. Both lead to inaccurate reports. Setting the right rules for each site takes a few minutes and makes every subsequent crawl much more reliable.
The most important crawl settings to configure
- URL exclusion rules: filter out /account/, /cart/, /checkout/, and any URL with session parameters like ?sessid= or ?ref=
- JavaScript rendering: turn on only for single-page apps (React, Angular, Vue) where content is added by scripts — it slows the crawl significantly
- Max pages: for very large sites, set a limit on the first crawl and increase it once you know the size
- Crawl speed: a slower crawl is polite on shared hosting; a faster one is fine on dedicated servers
When to crawl staging sites
- Before a site migration: catch redirect errors, broken links, and missing pages before launch (every step is in the SEO site migration checklist)
- After a theme or template change: verify that titles, descriptions, and canonical tags are still correct
- After a content migration from one CMS to another: confirm all old URLs redirect and no pages were lost
- Cloud crawlers cannot reach a staging URL that is only accessible on your own network — a desktop crawler can
Questions about crawl settings
Can a website send the crawler into my own network?
No. If a redirect or link sends SEOSage to an address inside a private network, such as your router or another device, SEOSage refuses to open it and shows a notice. A local or staging address that you start the crawl from is crawled as normal.
Is there a limit on how many pages I can crawl?
No. Max pages and Max depth both start at 0, which means no limit. You can set a limit for a site if you want one.
When should I turn on Render JavaScript?
Turn it on only if the website builds its content with scripts. If a page is only an empty app shell, SEOSage flags it as Content needs JavaScript to appear. It makes the crawl about four times slower and uses more memory, so leave it off for ordinary sites.
Can I crawl a staging site or a password-protected site?
Yes. Add a username and password in the project settings, and custom headers if the site needs them. If the site has a sign-in form, add the address of its login page and SEOSage signs in there before it crawls. They are stored on your computer only.
Can I pause a crawl, or carry on after closing the app?
Yes. Press Pause crawl next to Stop crawl and press Resume crawl when you are ready. A crawl also saves itself every two minutes, so if you stop it, close the app or the computer crashes, the project offers Resume crawl and carries on from the saved pages. A crawl that finishes clears its save.
Can I leave some pages out of a crawl?
Yes. Exclude URLs that match a pattern, for example /account/* or *?utm_*, and only include URLs that match a pattern if you want to crawl one section.
Can SEOSage crawl a site on a schedule?
Yes. Choose daily, weekly or monthly for each site. Scheduled crawls run while SEOSage is open or running in the menu bar.
Does it obey robots.txt?
By default, yes. You can turn Respect robots.txt off for a site you own and want to audit fully.
More SEOSage features
Every check and tool in the app, each with its own page.
SEO crawler White-label reports Competitor analysis Local site audit Link strength Custom extraction Security headers check Title tag width checker URL structure checker Page audit Change tracking Rank tracking AI Visibility Log analyser Core Web Vitals Compare crawls Backup and restore Google Analytics report Search Console report Robots.txt and sitemap audit Canonical tag checker Orphan pages Internal linking tool Keyword cannibalization Search Console quick wins Weekly SEO to-do list Fix and re-check Ask Claude All features
Set your own rules
Start your 14-day free trial and crawl the way you want. No card needed.