Robots.txt Generator

The Robots.txt Generator writes a robots.txt file for your site from your URL. It allows search crawlers, adds an explicit allow rule for each major AI crawler, and sets a Sitemap line pointing at your sitemap.xml. A few example Disallow lines for /api/, /admin/ and sort parameters are included. Delete the ones that do not apply, then upload the file to your site root.

Updated 23 October 2026

How to use the generated file

Enter your site address and the tool writes a robots.txt for it. The file has three parts: a rule for all crawlers, an allow rule for each major AI crawler, and a Sitemap line.

Save the result as robots.txt and upload it to the root of your site, so it loads at yourdomain.com/robots.txt. A robots.txt placed in a subfolder is ignored.

What is in the file

Each part of the output does one job.

Line	What it does
User-agent: * and Allow: /	Lets all crawlers reach the site
Disallow: /api/ and /admin/	Keeps those two folders out of crawls
Disallow: /?sort=	Skips URLs that carry a sort parameter
AI crawler block	An explicit allow rule for each AI crawler the tool tracks
Sitemap:	Points to yourdomain.com/sitemap.xml

Edit before you upload

The Disallow lines are examples. Your site may not have an /admin/ folder or a sort parameter, and it may have private areas the tool cannot know about.

OpenAI's training crawler is GPTBot, while OAI-SearchBot is the one ChatGPT search uses. If you want to be open to one and closed to the other, change the matching rule.

Delete the Disallow lines for folders you do not have.
Add a Disallow line for any area you want kept out of crawls, such as checkout or account pages.
Check that the Sitemap line points at your real sitemap address.
Upload the file to the root of your site and open it in a browser to confirm it loads.

Test the live file

After you upload it, run your site through the robots.txt checker. It parses every rule and flags a blanket block, unknown directives and a missing Sitemap line.

Common questions

Where does a robots.txt file go?

A robots.txt file goes at the root of your domain, so the address is yourdomain.com/robots.txt. Each subdomain needs its own file.

Does Disallow remove a page from Google?

Disallow doesn't remove a page from Google. It stops crawlers from fetching the page, but the address can still be listed if other sites link to it. To keep a page out of results, use a noindex tag and leave the page crawlable so the tag can be read.

Should I allow AI crawlers?

Whether to allow AI crawlers depends on whether you want AI engines reading your pages. Allowing them lets compliant crawlers fetch your content, and blocking them stops those crawlers. The generated file allows them.