Free tools / Robots.txt tester
Free robots.txt tester
Enter a website and a page path. See whether that page is allowed or blocked for a crawler, and which rule decided it.
What this tool checks
- Loads the robots.txt file from the website
- Tests any path for the crawler you choose
- Shows the rule that matched, using the longest-match rule Google uses
- Lists the groups of rules and the sitemaps declared in the file
- Warns when the whole website is blocked
Why it matters
A robots.txt file tells crawlers which parts of a website they may visit. One wrong line can hide a whole website from Google. This tool reads the live file and tests a path against it.
Want this for every page of a website? See Crawl controls in SEOSage.
How to read your result
- Allowed means the crawler may visit the path. Blocked means a rule stops it.
- The rule shown is the longest matching rule. When an Allow and a Disallow match equally, Allow wins.
- Rules are grouped by crawler. A crawler follows the most specific group that names it, and * covers the rest.
- The sitemaps listed come from Sitemap lines in the file.
Common problems and fixes
- Disallow: / for every crawler. This blocks the whole website. Remove it, or limit it to the folder you meant.
- A page you want found is blocked. Remove or narrow the Disallow rule. Blocked pages cannot be crawled.
- Blocking a page to keep it out of Google. Robots.txt stops crawling, not indexing. Use a noindex tag and allow crawling.
- No sitemap line. Add a Sitemap line so crawlers find your sitemap.
Questions about the robots.txt tester
What is a robots.txt file?
It is a small text file at the root of a website, for example example.com/robots.txt. It tells crawlers which paths they should not visit.
How do I know if a URL is blocked by robots.txt?
Enter the website, type the path, pick the crawler and press Test. The result says Allowed or Blocked and shows the rule that decided it.
What if there is no robots.txt file?
Then crawlers may visit everything. A missing file is not an error, but a site usually lists its sitemap there.
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked page can still appear in search without a description. To keep a page out, use a noindex tag and let crawlers visit it.
Does SEOSage check robots.txt?
Yes. Its crawl audits your robots.txt and declared sitemaps, and can skip or ignore robots.txt rules per site.
Check every page of your website
SEOSage crawls your whole website on your own computer, with no page limit, and lists what to fix. Free for 14 days, cancel any time.