The AI Crawler Checker reads the robots.txt at the root of your site and tests the major AI crawlers one by one, including GPTBot, ClaudeBot and PerplexityBot. It will also tell you if you have no robots.txt at all. Use it to find out whether AI engines can read your pages or not.
Updated
AI crawlers by default don't need a special invitation to read a public site, but in some cases your robots.txt can actually block them. Usually if bots are blocked like that, it's an intentional thing, but sometimes it can happen by accident when a security product, CDN, hosting template, or old privacy change adds rules to your site without you knowing about it. That latter unintentional case is what this tool is designed to detect.
This AI crawler checker reads the robots.txt file at the root of your site and evaluates the major AI user agents (GPTBot, ClaudeBot and PerplexityBot) separately. It reports allowed, blocked, and contradictory instructions, or if you have no robots.txt at all. Contradictory instructions means your robots.txt has multiple rules conflicting; not every crawler resolves the conflict the same way, so it's best to just remove or resolve the conflicts in the file. If you want to show up in AI search, make sure the search crawlers aren't blocked: OAI-SearchBot, Claude-SearchBot, ChatGPT-User and PerplexityBot. Claude-SearchBot is the one that matters for Claude's search answers. A crawler with no rule is allowed by default.
| Crawler | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content to train OpenAI's models |
| OAI-SearchBot | OpenAI | Finds pages to show and cite in ChatGPT search |
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT user asks it to |
| ClaudeBot | Anthropic | Collects content to train Claude |
| Claude-SearchBot | Anthropic | Finds pages to show and cite in Claude's search answers |
| Claude-User | Anthropic | Visits a page when a Claude user asks it to |
| PerplexityBot | Perplexity | Indexes pages for Perplexity's answers. Perplexity says it isn't used for training. |
| Google-Extended | Controls whether Gemini can use your content. AI Overviews and AI Mode use Googlebot, so this doesn't affect them. | |
| CCBot | Common Crawl | Builds an open web archive that many AI models are trained on |
| Applebot-Extended | Apple | Controls whether Apple can use your content to train its models |
The "User" crawlers only visit when a person asks about a specific page. Some companies say robots.txt may not apply to these visits.
Allowing GPTBot or ClaudeBot does not guarantee that ChatGPT or Claude will quote you. It only removes one mechanical reason they cannot read the page directly. The page still needs a clear answer, credible evidence, useful structure, and enough authority to be selected.
If an important crawler is blocked, new pages may be described from search snippets, third-party mentions, or older material.
This AI crawler checker reads your robots.txt. A CDN or firewall can still block AI crawlers even when robots.txt allows them. Cloudflare blocks known AI crawlers by default on new domains. If you use a firewall or a CDN like Cloudflare, check its bot settings too.
Open the robots.txt shown in the result and find the named user-agent block. A line such as Disallow: / prevents that crawler from reading the entire site. Remove the block only if it matches your policy.
This lets AI search engines read and cite your site and keeps your content out of model training. Copy it into your robots.txt and change it to fit your policy.
# AI search and user visits: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
Allow: /
Disallow: /account/
# Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Disallow: /A crawler named in its own group ignores your User-agent: * rules. Repeat any private paths in that group, like /account/ above.
Crawler names represent different products and purposes. GPTBot is OpenAI's training crawler, and ChatGPT search uses OAI-SearchBot. ClaudeBot is Anthropic's training crawler, and Claude's search answers use Claude-SearchBot. Google-Extended is a robots.txt token and does not control AI Overviews.
Decide what you want to permit, then write an explicit rule for each crawler.
Once the crawlers can read your site, you've cleared the main technical hurdle, but now you want to make sure your content and pages are optimized for AI engines to read them.
You can try the Free AEO Checker as a good next step; this scores a single page on how easy it is to quote.
You can also add an llms.txt, a short summary of your site’s key pages written for AI tools. It’s a proposed standard and none of the major AI engines have confirmed they use it yet, so treat it as optional. The free llms.txt generator makes one in a minute if you want it. The llms.txt checker tests the file against the current spec.
Allowing AI crawlers is just the first step to getting cited. Actually getting quoted depends on a variety of factors, mainly your overall domain reputation and how well the content answers the question. Check whether your pages answer the question directly but comprehensively near the top, and whether the engines are citing third-party articles, listicles and roundups rather than first-party blogs and content pages. If the latter is true, it will be difficult to get mentioned in the answer without finding a way to get mentioned or featured on those 3rd party sites.
In Peak Answer's research, the specificity of the question you're trying to get quoted for also mattered quite a bit, so consider that as well.
Peak Answer runs checks like these every week, then writes the content and gets the placements.