The Robots.txt Checker fetches the robots.txt file at the root of your domain and runs 6 checks on it: whether the file loads and parses, whether a sitemap is declared, whether major AI crawlers are blocked, whether the whole site is blocked, whether any lines are unsupported, and whether it's under Google's 500 KiB limit. It also prints the file.
Updated
| Check | Passes when | Warns or fails when |
|---|---|---|
| File loads and parses | /robots.txt returns 200 with at least one User-agent line | It's missing, returns an error, or has no User-agent lines |
| Sitemap declared | There's a Sitemap line with a full URL | There's no Sitemap line |
| AI crawlers allowed | No major AI crawler is blocked from the whole site | One or more is blocked. The result names each one |
| No blanket block | The User-agent: * group doesn't say Disallow: / | It does, so every crawler that follows the file is kept out of the whole site |
| No unsupported directives | Every line uses a directive crawlers read | A line uses something crawlers ignore, like noindex |
| Size under 500 KiB | The file is under 500 KiB | It's larger. Google ignores everything past that point |
The headline says the file is clean or counts the things to fix. Each check comes back as a pass, a warning or a fail, and the file is printed underneath so you can find the lines behind each one.
A missing file (a 404) is safe. Crawlers treat the site as open, so nothing is blocked, but you lose the chance to point them at your sitemap. A server error (5xx) is serious. Google stops crawling the site for the first 12 hours of the error, then falls back to the last copy it had.
| Response for /robots.txt | What Google does |
|---|---|
| 200 | Reads the file and follows its rules |
| Redirect (3xx) | Follows at least 5 redirects, then treats the file as missing |
| 404 or another 4xx | Treats the site as having no robots.txt and crawls everything |
| 429 or 5xx | Stops crawling the site for the first 12 hours and keeps trying to fetch the file. After that it uses the last copy it had for up to 30 days, or crawls freely if it has none. If the errors last past 30 days, it crawls as if there were no file, unless the site is unavailable in general |
Source: Google's robots.txt specification.
| Line | What it does | Notes |
|---|---|---|
User-agent | Starts a group of rules for one crawler, or * for all of them | A crawler follows only the most specific group that matches its name. Names aren't case-sensitive |
Disallow | Blocks a path | Paths are case-sensitive. Disallow: with nothing after it blocks nothing |
Allow | Opens a path inside a blocked one | Google and Bing both read it |
Sitemap | Points crawlers to a sitemap | Use the full URL. You can list several, anywhere in the file |
Crawl-delay | Asks a crawler to wait between requests | Google ignores it. Bing reads it |
Noindex | Nothing | Google stopped reading it in 2019. Use a noindex meta tag |
* and $ | * matches any characters, $ marks the end of a URL | Disallow: /*.pdf$ blocks every PDF |
When an Allow and a Disallow both match a URL, Google uses the rule with the longer path. If they're the same length, Allow wins. Other crawlers may settle it differently, so avoid rules that overlap.
A typical site. Crawlers can read everything except admin, cart and internal search pages.
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Sitemap: https://www.example.com/sitemap.xmlBlock one crawler, allow the rest.
User-agent: *
Disallow: /admin/
User-agent: GPTBot
Disallow: /
Sitemap: https://www.example.com/sitemap.xmlGPTBot reads only its own group, so the /admin/ rule doesn't apply to it. Here that's fine, since it's blocked from everything. If you give a crawler its own group and still want private paths closed, repeat them in that group.
A staging site.
User-agent: *
Disallow: /Use this only on a staging or test site. On a live site it hides every page. A password protects a staging site better, since a blocked URL can still show up in search results if other sites link to it.
For AI crawlers, the AI Crawler Checker has an example that allows AI search and blocks model training.
| Method | Stops crawling | Keeps the page out of search results |
|---|---|---|
| robots.txt Disallow | Yes | No. The URL can still appear if other pages link to it |
| noindex meta tag or X-Robots-Tag header | No | Yes, once the page is crawled. Crawlers have to be allowed to fetch the page to see the tag |
| Password or login | Yes | Yes |
Don't block a page in robots.txt and add noindex to it. The crawler can't fetch the page, so it never sees the noindex.
Disallow: / from staging. The site launches and stays hidden.| Platform | Where to edit it |
|---|---|
| WordPress | WordPress serves a virtual file by default. Edit it with an SEO plugin's file editor (Yoast SEO or Rank Math), or upload your own robots.txt to the site root |
| Shopify | Edit the robots.txt.liquid theme template. Shopify's guide has the steps |
| Wix | The Robots.txt Editor, under Tools and settings in the SEO & GEO dashboard. Wix's guide has the steps |
| Squarespace | Squarespace generates the file and doesn't let you edit it. Its help center says so |
| Next.js | Add app/robots.ts, or a static app/robots.txt. The Next.js docs show both |
The AI crawlers check fails when a major AI crawler is blocked from the whole site, and it names the ones it found. Each AI company usually runs more than one crawler. GPTBot is OpenAI's training crawler and ChatGPT search uses OAI-SearchBot, so a rule for one doesn't cover the other. Google-Extended is a robots.txt token and doesn't control AI Overviews.
For a per-crawler result and the full list tested, use the AI Crawler Checker.
Disallow: / under User-agent: * if you want the site crawled.Peak Answer runs checks like these every week, then writes the content and gets the placements.