Free tools / Robots.txt tester
Free robots.txt tester
Enter a website and a page path. See whether that page is allowed or blocked for a crawler, and which rule decided it.
See the free robots.txt tester in 22 seconds
Enter a website and a page path. See whether the page is allowed or blocked for a crawler, and which rule decided it. The result in the video is a real run.
Read the video transcript
Is this page blocked from Google? Enter a website and a page path. Pick a crawler, and press Test robots.txt. It shows if the page is allowed or blocked, the rule that decided it, and the whole file. That is one page. SEOSage checks every page of your website. Start your fourteen-day free trial at seosage.co.
What this tool checks
- Loads the robots.txt file from the website
- Tests any path for the crawler you choose
- Shows the rule that matched, using the longest-match rule Google uses
- Lists the groups of rules and the sitemaps declared in the file
- Warns when the whole website is blocked
Why it matters
A robots.txt file tells crawlers which parts of a website they may visit. One wrong line can hide a whole website from Google. This tool reads the live file and tests a path against it.
Want this for every page of a website? See Crawl controls in SEOSage.
How to read your result
- Allowed means the crawler may visit the path. Blocked means a rule stops it.
- The rule shown is the longest matching rule. When an Allow and a Disallow match equally, Allow wins.
- Rules are grouped by crawler. A crawler follows the most specific group that names it, and * covers the rest.
- The sitemaps listed come from Sitemap lines in the file.
Common problems and fixes
- Disallow: / for every crawler. This blocks the whole website. Remove it, or limit it to the folder you meant.
- A page you want found is blocked. Remove or narrow the Disallow rule. Blocked pages cannot be crawled.
- Blocking a page to keep it out of Google. Robots.txt stops crawling, not indexing. Use a noindex tag and allow crawling.
- No sitemap line. Add a Sitemap line so crawlers find your sitemap.
What is a robots.txt tester?
A robots.txt tester is a tool that reads a website's robots.txt file and shows whether a specific URL is allowed or blocked for a given crawler. Every website can have a robots.txt file at its root (for example, example.com/robots.txt). The file uses a simple format of User-agent and Disallow or Allow rules to tell Googlebot, Bingbot, GPTBot, ClaudeBot and other crawlers which paths they may visit. One wrong Disallow rule can block an entire website from being crawled.
This free tool reads the live robots.txt file from the domain you enter, then evaluates the path you test using the longest-match rule that Googlebot follows. It shows which rule matched and whether the result is Allowed or Blocked. You can switch the crawler (Googlebot, Bingbot, GPTBot, ClaudeBot or the wildcard) to see how different bots are treated. The whole-site version of this check is built into SEOSage: a crawl audits your robots.txt and flags any important pages that are blocked.
What robots.txt can and cannot do
- It stops crawling, not indexing. A blocked page can still appear in Google if another site links to it, without a snippet. To remove a page, use a noindex tag and allow crawling so Google can read the tag.
- It does not apply equally to all crawlers. Each bot reads only the group that names it, or the wildcard group. Blocking Googlebot does not block Bingbot.
- AI crawlers respect it too. GPTBot, ClaudeBot, PerplexityBot and similar crawlers follow robots.txt. Use a User-agent group to allow or block them separately.
- A missing robots.txt is fine. Crawlers treat it as open access. A missing file is not an error, but the file is a good place to declare your sitemap.
Questions about the robots.txt tester
What is a robots.txt file?
It is a small text file at the root of a website, such as example.com/robots.txt. It tells crawlers which paths they should not visit.
How do I know if a URL is blocked by robots.txt?
Enter the website, type the path, pick the crawler and press Test. The result says Allowed or Blocked and shows the rule that decided it.
What if there is no robots.txt file?
Then crawlers may visit everything. A missing file is not an error, but a site usually lists its sitemap there.
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. A blocked page can still appear in search without a description. To keep a page out, use a noindex tag and let crawlers visit it.
Does SEOSage check robots.txt?
Yes. Its crawl audits your robots.txt and declared sitemaps, and can skip or ignore robots.txt rules per site.
Check every page of your website
SEOSage crawls your whole website on your own computer, with no page limit, and lists what to fix. Free for 14 days, cancel any time.