Which AI Crawlers Do the Sites ChatGPT and Google Cite Block?

What the numbers show

Among ChatGPT-cited domains, GPTBot was blocked on 9 of 394 (2.3%) and OAI-SearchBot on 4 of 394 (1.0%).

15 of 784 domains (1.9%) blocked GPTBot but allowed OAI-SearchBot.

Of 1540 cited URLs, 1290 (83.8%) returned a real page, 123 (8.0%) returned a bot challenge and 98 (6.4%) were skipped because the site's robots.txt disallows Peak Answer's crawler. Of 1290 real pages, 72 (5.6%) were near-empty without JavaScript. A sitemap was found on 771 of 820 domains (94.0%).

The sample is sites cited for tech-sector questions (T01 to T15). The study is descriptive only.

Method

Peak Answer took 1540 unique cited URLs on 847 domains and 887 hosts from tech-sector questions (T01 to T15). ChatGPT citations came from DataForSEO's ChatGPT scraper (cited pages only). Google organic and AI Overview citations came from earlier SERP data.

Each host's own robots.txt was fetched and parsed with grouped user-agent blocks, longest-match precedence with Allow winning ties, and wildcards. Domain-level figures test the domain's first cited URL path. URL-level figures use all cited URLs. Blocked means the cited path is disallowed for that agent. Blanket means the whole site is disallowed for that agent. No robots.txt counts as allowed and is shown separately.

Pages were fetched with a plain request, no JavaScript. A page is near-empty when it has under 150 visible words and at least one external script. Bot-challenge pages are their own category. ChatGPT-cited domains were cited at least once by ChatGPT. The other domains were cited by Google organic or AI Overview and not by ChatGPT. First-party is a small provider's own site and giant is a large vendor.

Limits

• robots.txt, llms.txt and page content were read on the day of the check (October 2026), not on the day each URL was cited. Rules may have changed in between.
• 636 of 887 hosts and 979 of 1540 URLs were measured the day before and reused. Just 251 hosts and 561 URLs were fetched fresh, so a change in that one day is not reflected for reused hosts.
• Pages were fetched with a plain request and no JavaScript, with requests to each host spaced out and a 10 second timeout.
• A cited page is skipped when Peak Answer's own crawler is disallowed by that site's robots.txt.
• Denominators are domains whose robots.txt was readable or absent. No robots.txt counts as allowed and is shown separately.
• Cells with a denominator under 10 are marked.
• Bot-challenge pages are their own category and are not counted as real pages.
• Near-empty means under 150 visible words with at least one external script.
• llms.txt presence was recorded for each host, but the check cannot tell a real llms.txt file from other plain-text responses, so no adoption rate is given.
• The ChatGPT-cited set grew to 402 domains from 248 in the first pass, because the scraper returns about 6.3 cited pages per answer against about 2.9.

Download the data

The CSV files hold the figures behind the tables. Units are in the column names.

• Share of domains blocking each crawler, by segment (80 rows)
• What happened when the cited pages were fetched (6 rows)
• robots.txt state and sitemaps by domain segment (10 rows)

How to cite this study

Peak Answer, "Which AI Crawlers Do Cited Sites Block?", https://peakanswer.com/research/ai-crawler-access.

Related

The AI crawler checker looks at crawler access for one site.

The llms.txt generator makes an llms.txt file.