Search tools

Search for a command to run...

Robots.txt Generator (incl. AI Crawlers)

Create a robots.txt file — allow or block search engines, disallow private folders, block AI training crawlers like GPTBot and ClaudeBot, and add your sitemap.

How to use

  1. 1Choose the default rule, folders to block and whether to block AI crawlers.
  2. 2Save the result as robots.txt in your site's root (example.com/robots.txt).

Frequently asked questions

Does robots.txt hide pages from Google?

It stops crawling, not indexing — a blocked URL can still appear if linked elsewhere. Use a noindex meta tag to keep a page out of results.

Can I block AI training bots?

Yes — major AI crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, Common Crawl, Google-Extended and others) honor robots.txt rules. Not every scraper does.

Where does the file go?

At the root of the domain, e.g. https://example.com/robots.txt. Subdomains need their own file.