Robots.txt Generator for AI Crawlers

Pick Allow or Block for each of the 11 artificial intelligence (AI) crawlers below, add any private paths, then upload the file as robots.txt at your site root. At Layer3Labs, we run technical search engine optimization (SEO) audits across the sites we build and operate. The default setup allows search and user-fetch crawlers alongside Googlebot and Bingbot, while blocking GPTBot, ClaudeBot, Google-Extended, and CCBot. Selecting Block for Google-Extended opts your content out of Gemini foundation model training and Gemini grounding without affecting your inclusion or ranking in regular Google Search.

This generator handles group inheritance automatically. Under the Robots Exclusion Protocol (RFC 9309), a crawler that finds a group naming it ignores the * group. The Robots.txt Generator therefore repeats your private-path Disallow lines in the group for allowed crawlers, so OAI-SearchBot and the other named crawlers that follow robots.txt still skip those paths.

The table What Each Crawler Will Do With This File, under the generated file, shows how each crawler reads it. Once you verify your choices, copy the text or download the file and upload it directly to the root directory of your web server as a UTF-8 (Unicode) encoded text document.

Robots.txt Generator

User-agent: *
Allow: /

User-agent: OAI-SearchBot
User-agent: Googlebot
User-agent: Bingbot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

How to Set Up Robots.txt for AI Crawlers

You set up robots.txt for AI crawlers by giving each choice its own user-agent group in one file at your site root.

Follow these steps to generate and deploy your file:

• 1. Start from a preset: Choose from the three options in the generator to set default permissions across search, user-fetch, and training bots.
• 2. Adjust individual crawlers: Use the Allow/Block menu beside each crawler. Changing any one switches the preset to Custom.
• 3. Add private paths: Enter any directory or file paths you want to keep private, entering one path per line starting with a forward slash.
• 4. Add your sitemap URL: Enter the complete web address (uniform resource locator, URL) for your Extensible Markup Language (XML) sitemap.
• 5. Save and deploy: Select Copy robots.txt or Download robots.txt, save the file as a UTF-8 text file named robots.txt, and place it at the root of your domain host.
• 6. Check it live: Run the AI Crawler Checker. The AI Crawler Checker requests your pages directly, so it also catches a firewall or content delivery network (CDN) that refuses a crawler your robots.txt allows.

Choosing a Preset for Your Crawling Strategy

The generator includes three presets to quickly configure common crawling policies.

The default preset, Allow AI search, block AI training, permits search bots and user-directed fetchers while blocking training bots. It sets Allow for OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot, ChatGPT-User, and Claude-User. It sets Block for GPTBot, ClaudeBot, Google-Extended, and CCBot. According to Google documentation on common crawlers, Google-Extended is a standalone product token that controls whether site content is used for Gemini training and grounding. Blocking Google-Extended does not affect your inclusion in Google Search and is not a ranking signal. See what each crawler does in the AI Crawler Checker.

The second preset, Allow every AI crawler, sets every crawler to Allow.

The third preset, Block every AI crawler (keep Google and Bing Search), blocks all AI search, user-fetch, and training bots while leaving Googlebot and Bingbot set to Allow. Pick it if you want Google and Bing search traffic but want the other AI search, user-fetch and training crawlers in the list kept out. OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may reduce visibility in Claude's search results.

Reading the File the Generator Writes

The generator organizes rules into structured blocks to ensure every bot receives explicit instructions. In standard robots.txt files, rules are grouped under User-agent lines. Per Google documentation on creating a robots.txt file, a crawler matches only one rule set: the most specific group that matches its user agent. The standard wildcard token matches all crawlers except Google AdsBot crawlers, which require explicit naming.

When you load the generator with default settings, it produces three distinct groups followed by an optional sitemap line. The first group is User-agent: * with Allow: /.

The second group explicitly names every allowed AI search and user-fetch bot alongside traditional search crawlers. This group lists User-agent: OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User, then Allow: /.

The third group lists GPTBot, ClaudeBot, Google-Extended and CCBot, then Disallow: /.

In our internal analysis, we re-read three robots.txt layouts with the same parser the AI crawler checker uses. Layout A represents a common hand-edited file where an author defines Disallow: /admin/ under the wildcard group, creates a separate group for OAI-SearchBot and PerplexityBot with Allow: /, and blocks GPTBot. Under RFC 9309, the Robots Exclusion Protocol standard, because OAI-SearchBot and PerplexityBot match their own specific group, they completely ignore the wildcard group. As a result, OAI-SearchBot and PerplexityBot can crawl /admin/login, Bingbot follows the wildcard group and is blocked from /admin/, and GPTBot is blocked everywhere.

Layout B demonstrates how this generator structures rules. It repeats the Disallow: /admin/ line inside the named group for allowed crawlers. In this layout, OAI-SearchBot and PerplexityBot are blocked from /admin/login while still able to fetch the home page. With no private paths, giving OAI-SearchBot its own Allow group changes nothing. The gap appears only once you add private paths under the * group. You can check this in the What Each Crawler Will Do With This File table by adding a private path.

Path and Sitemap Syntax Rules

Every path entered into the private paths field must begin with a forward slash. Rules are case-sensitive, per Google's robots.txt guide. A rule targeting /private.html will not match requests for /PRIVATE.html or /Private.html.

Sitemap declarations must point to a complete web address. Under Google robots.txt guidelines, the Sitemap directive requires a fully qualified URL including the protocol, host, and path. The generator validates your sitemap input and will omit the line if you provide an incomplete or relative path.

Keep in mind that robots.txt is a publicly viewable plain-text document, not an access control system. Anyone can inspect your file to see the directory paths you have listed. If a path contains proprietary files or sensitive administrative tools, protect that content using server-level authentication or login credentials rather than relying on robots.txt disallow rules.

Our Takeaway on Structuring Robots.txt

For most business sites that want AI search answers to cite them, allow the AI search and user-fetch crawlers and keep Googlebot and Bingbot open. Block GPTBot, ClaudeBot and CCBot, and decide Google-Extended knowingly. Blocking Google-Extended opts your site out of Gemini model training and grounding, does not affect inclusion in Google Search, and is not a ranking signal. Always verify that private paths are repeated across every named user-agent block, and test your live server using our AI Crawler Checker.

It does not fit a site that wants to stay out of AI search entirely. Pick Block every AI crawler (keep Google and Bing Search) to leave ChatGPT search answers and possibly lose visibility in Claude's search results. Googlebot and Bingbot stay allowed in that preset, so Google's and Bing's own AI answers can still read your pages. It also does not serve a site that needs access control. Put confidential documents behind a login, because anyone can read robots.txt.

Our guidance would change if a vendor merged search and training into one crawler token. It would also change if a new training opt-out signal replaced per-crawler rules. To structure the rest of your machine-readable content, use our llms.txt Generator or test your general search discovery using our AI Visibility Checker.