Free tools / llms.txt checker
Free llms.txt checker
Enter a website. We read its llms.txt file, test the links in it, and show which AI crawlers its robots.txt lets in.
What this tool checks
- Finds the llms.txt file at the root of the website
- Checks the format: one # title, a > summary and ## sections of links
- Tests the links in the file, 15 at a time, for errors and redirects
- Spots a web page served in place of the file
- Reads robots.txt for 11 AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot
- Notes whether an llms-full.txt file exists too
Why it matters
llms.txt is a proposed standard from 2024: a short Markdown file that points AI tools to the most useful pages of a website. Support is still limited, and Google has said its search does not use it. Whether AI crawlers may read your pages at all is set by robots.txt, so this tool checks both.
Source: llmstxt.org: the llms.txt proposal.
How to read your result
- Fix means the file breaks the format or a link is broken. Check means it is worth a look. Info is just a fact.
- Links are tested 15 at a time, in the order they appear. Press the button under the table for the next batch.
- Allowed means robots.txt lets that crawler visit your home page. Blocked means a rule stops it, which can be on purpose.
- Google-Extended and Applebot-Extended are names used only in robots.txt. Blocking them stops AI use of your content, not your normal search listing.
Common problems and fixes
- A web page instead of the file. Your server returns a normal page at /llms.txt. Upload a plain text file with that name to the root of the website.
- No # title line. Start the file with one line such as # Your Site Name.
- Broken links in the file. Point each link at a working page, using its final address.
- AI crawlers blocked by accident. A Disallow: / under User-agent: * blocks AI crawlers too. Add a group for each crawler you want to let in.
What is an llms.txt checker?
An llms.txt checker is a tool that loads the llms.txt file at the root of a website and checks it against the format proposed at llmstxt.org: one # title line with the site name, an optional > summary, and ## sections that list links in Markdown. It also tests that each link in the file works. This checker reads the robots.txt file as well, because robots.txt, not llms.txt, decides whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot may visit the pages at all.
Questions about the llms.txt checker
What is an llms.txt file?
A plain text file, written in Markdown, at yoursite.com/llms.txt. It gives AI tools a short summary of the website and links to its most useful pages. It was proposed at llmstxt.org in 2024.
Does llms.txt help SEO?
Not in Google. Google has said its search does not use llms.txt, and there is little public proof that AI assistants read it. It takes minutes to add, so some sites add it in case support grows.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which pages they may visit. llms.txt only suggests what to read. If robots.txt blocks a crawler, llms.txt does not let it in.
What should an llms.txt file contain?
One line that starts with # and the site name, then one line that starts with > and sums up the site, then ## sections that list links as - [Page name](address): short note. A section called Optional holds links that can be skipped.
Does blocking Google-Extended remove my site from Google?
No. Google-Extended only controls whether Google may use your content for its Gemini AI. Google Search uses Googlebot, which has its own rules.
Check every page of your website
SEOSage crawls your whole website on your own computer, with no page limit, and lists what to fix. Free for 14 days, cancel any time.