Skip to content
SEOSage

Features / Crawl controls

Crawl settings you control, for every website

SEOSage has no hidden limits. Set the pages, speed, JavaScript and sign-in rules for each site, and crawl staging sites before they go live.

Download SEOSage

Jump to: What to crawl · How to crawl · Staging sites · Questions

Decide what gets crawled

Every website is different. Set the rules for each site in its project settings, before you start.

  • No page limit: Max pages and Max depth both start at 0, which means no limit
  • Include subdomains such as blog. or shop.
  • Leave out URLs with a pattern, for example /account/* or *?utm_*
  • Treat ?ref= variants of a page as one URL
  • Add extra start URLs, and your sitemap and robots.txt addresses
SEOSage
Project settingsmarketingly.org

Scope

Crawl subdomainsInclude blog., shop., etc.
Remove URL parametersTreat ?ref= variants as one URL
Exclude URLs matchingOne pattern per line. Use * as a wildcard./account/*
*?utm_*

Authentication

For crawling password-protected or staging sites. Stored locally on this device.

UsernamePassword

Custom headers

X-Bypass-Token: abc123

Schedule

OffDailyWeeklyMonthly
rules for each site

Decide how it crawls

Choose how SEOSage behaves while it crawls, and how fast.

  • Crawl as the SEOSage desktop bot, or another user agent
  • Render JavaScript, to see content that scripts add to the page. It makes a crawl about four times slower, so leave it off unless you need it
  • Respect robots.txt, or turn it off
  • Follow nofollow links, and check the status of external links
  • Set the crawl speed in URLs per second
SEOSage
Project settingsmarketingly.org

Crawl

Crawl asThe bot / user agent SEOSage identifies asSEOSage (Desktop)
Render JavaScriptMakes the crawl about 4 times slower. Turn on only if the site builds its content with scripts.
Respect robots.txtSkip pages disallowed for crawlers
Follow nofollow links
Check external linksVerify the status of outbound links

Limits

Crawl speedMax URLs fetched per second8
Max pages0 = no limit0
Max depthClicks from the start URL · 0 = no limit0
one dialog per site

Crawl sites that are not public

Because SEOSage runs on your own computer, you can audit a website before it goes live.

Password-protected and staging sites

  • Add a username and password. They are stored on your computer only
  • Send custom headers with every request, one per line

Repeat crawls

  • Schedule a site to be crawled daily, weekly or monthly
  • Scheduled crawls run while SEOSage is open or running in the menu bar
  • Stop a crawl any time and keep the results so far

See how the results look on the SEO crawler page, or test a robots.txt rule with the free robots.txt tester.

Questions about crawl settings

Is there a limit on how many pages I can crawl?

No. Max pages and Max depth both start at 0, which means no limit. You can set a limit for a site if you want one.

When should I turn on Render JavaScript?

Turn it on only if the website builds its content with scripts. It makes the crawl about four times slower and uses more memory, so leave it off for ordinary sites.

Can I crawl a staging site or a password-protected site?

Yes. Add a username and password in the project settings, and custom headers if the site needs them. They are stored on your computer only.

Can I leave some pages out of a crawl?

Yes. Exclude URLs that match a pattern, for example /account/* or *?utm_*, and only include URLs that match a pattern if you want to crawl one section.

Can SEOSage crawl a site on a schedule?

Yes. Choose daily, weekly or monthly for each site. Scheduled crawls run while SEOSage is open or running in the menu bar.

Does it obey robots.txt?

By default, yes. You can turn Respect robots.txt off for a site you own and want to audit fully.

Set your own rules

Download SEOSage for Mac or Windows and crawl the way you want. Free for 14 days, cancel any time.

Download SEOSage