Free AI Crawler Checker for GPTBot, ClaudeBot and More

Check Which AI Crawlers Can Reach Your Site

To see which artificial intelligence crawlers can access your pages, paste your page address into the input field above and click Check AI Crawler Access. At Layer3Labs, we run SEO and AI-search routines across the sites in our portfolio. The free AI crawler checker checks eleven distinct bots two ways, testing your published robots.txt file and making a live server request using each crawler's published user agent. If you want a complete audit of your site's readiness for conversational search engines, our free AI visibility checker scores your page out of 100 across crawler access, raw text rendering without JavaScript, structured schema data, and answer formatting.

When you enter a page address into the field marked Page to check (your homepage or any URL) and click the button, our server requests your page once as a regular browser visitor. It follows redirects to the final host. It reads /robots.txt from that host and requests the page once per crawler with that crawler's user agent. The button reads Testing 11 crawlers while this runs.

The audit delivers an immediate overview banner alongside the AI Crawler Access, Bot by Bot table. This output details the exact directive that governs each crawler, displays live response codes, and provides copyable robots.txt rules if robots.txt blocks a search crawler.

Reading the Crawler Table

The top of the test report presents a clear status headline stating either Every AI search crawler can reach this page or N AI search crawlers cannot reach this page. Below this summary, the AI Crawler Access, Bot by Bot table organizes findings into five columns: Crawler, Type, robots.txt, Test request, and Result. The Crawler column names the bot token, its operating organization, and its operational purpose. The Type column categorizes the crawler into Search / citations, User-requested fetch, or Training. The robots.txt column displays Allowed or Blocked alongside the exact line that determined the outcome and the user-agent group that applied, whether an asterisk wildcard or the crawler's specific name. The Test request column logs the HTTP response status code, displaying values like 200, 403, or Not sent.

Every row in the table concludes with one of five specific Result labels. Can reach the page shows as green, signifying that robots.txt allows access and the server did not refuse the test request. Blocked by robots.txt indicates that your directives explicitly forbid crawling. Refused by the server marks a configuration where robots.txt permits crawling, but the live server request returned an error code like 403 or 429 while normal visitors succeeded. Blocked by robots.txt and refused shows that both the text directives and the live web server rejected the crawler. No answer to the test indicates that the crawler probe received no response, or that your web server failed to respond even to the baseline standard visitor request.

Result color codes distinguish between indexation risks and deliberate data exclusions. Blocked search crawlers receive a red indicator because excluding search bots prevents your content from being retrieved or cited in live conversational answers. Blocked training or user-fetch crawlers show a yellow indicator because blocking model training or live user fetches is often an intentional editorial decision. If your test uncovers blocked search crawlers, a copyable text box appears with pre-formatted Allow directives, accompanied by a Copy robots.txt Lines button. You can also click Copy Report to place the complete row-by-row audit onto your clipboard, or use the share link to re-run the live test with coworkers.

A clean evaluation produces all green rows across the table. For example, when we audited our own homepage on 25 October 2026, the robots.txt file returned HTTP 200, assigning all crawlers to the User-agent: * group with Allow: /. All ten live probe requests returned HTTP 200, while Google-Extended was marked Not sent. The test generated the headline Every AI search crawler can reach this page, with all eleven rows displaying Can reach the page, requiring no copyable fix lines or firewall warnings. Conversely, if a domain lists User-agent: OAI-SearchBot followed by Disallow: /, that specific row displays Blocked by robots.txt in red, citing that rule line in the robots.txt column.

Search, User-Requested, and Training Crawlers

AI providers deploy specialized bots for distinct functions, separating search indexation from offline machine learning. Search crawlers collect content for retrieval engines that quote websites in live answers. OpenAI documents on its bot overview page that OAI-SearchBot surfaces websites directly within ChatGPT search features. Sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers, although they can still show up as navigational links. Similarly, Anthropic notes in its crawler documentation that Claude-SearchBot indexes web content to improve search responses, and blocking it may reduce how often a site appears in search results. Perplexity explains in its bot guide that PerplexityBot indexes websites for search results and citations without training foundation models. Googlebot and Bingbot are also typed as search crawlers. Google requires a page to be indexed in Google Search before it can appear as a supporting link in Google AI Overviews.

User-requested crawlers fetch URLs only when an individual enters a prompt requiring immediate external browsing. OpenAI documents that ChatGPT-User handles these real-time visits, adding that standard robots.txt directives may not always apply to direct user navigation requests. Anthropic states that Claude-User fetches pages on demand whenever a Claude user submits a prompt that explicitly requires browsing external sites. Perplexity documents that Perplexity-User handles immediate queries and generally ignores robots.txt directives.

Training crawlers harvest public web data to build future machine learning models. OpenAI documents that GPTBot collects content that may be used to train OpenAI foundation models, noting that blocking GPTBot does not impact search appearance because each setting operates independently. Anthropic documents that ClaudeBot gathers web data for training models rather than indexing content for search answers. Google notes in its crawling documentation that Google-Extended is a standalone robots.txt token that controls whether content Google crawls may be used for training future Gemini models and for grounding. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Common Crawl runs CCBot, which identifies itself as CCBot/2.0 and builds an open dataset used for training. Because blocking a training crawler is often a deliberate choice, our checker flags blocked training crawlers in yellow rather than red.

Robots.txt Evaluation and RFC 9309 Rules

Our AI crawler checker implements the standard Robots Exclusion Protocol defined in RFC 9309. Under these rules, specific user-agent declarations override generic wildcard blocks. When the AI crawler checker parses your robots.txt file, it checks whether a crawler has a named User-agent block that matches its identifier. If a specific block exists, the crawler evaluates only the directives inside that group, ignoring all rules declared under User-agent: *. If no bot-specific block exists, the crawler falls back to the rules defined in the wildcard group. If a domain serves no robots.txt file at all, the standard treats every bot as fully allowed.

Rule precedence within a group follows path length and specificity. When multiple directives match a requested URL, the longest matching pattern determines the outcome. If an Allow rule and a Disallow rule share the exact same character length on a matching path, the checker applies the Allow directive, as RFC 9309 says. If your web server experiences an outage and serves a 5xx HTTP server error when crawlers request /robots.txt, or if the file times out completely, RFC 9309 says crawlers should assume full disallow until the file is restored.

This hierarchy allows operators to permit AI search indexation while excluding training crawlers on the same domain. Consider the following robots.txt configuration:

• User-agent: * then Allow: /
• User-agent: GPTBot then Disallow: /
• User-agent: OAI-SearchBot then Allow: /

Server Refusals and Firewall Detection

Permissive rules in robots.txt do not guarantee that an automated crawler can fetch your pages. A crawler is marked Refused by the server whenever its specific probe request receives a 401, 402, 403, 429, or 503 response while our baseline browser request to the exact same URL receives a successful 200 response. The usual cause is a firewall, CDN or hosting bot rule.

Cloudflare announced on 1 July 2025 that blocking AI crawlers became the default setting for its customers, while leaving site administrators the ability to manually allow approved bots. When managed bot protections or web application firewalls are enabled, automated requests carrying strings like OAI-SearchBot or Claude-SearchBot can be blocked at the edge. If your robots.txt file permits these bots but your test request returns Refused by the server, check your firewall, CDN or hosting bot settings.

The AI crawler checker sends published user agent strings for OpenAI bots, Googlebot, Bingbot, PerplexityBot, and CCBot, while Anthropic crawlers use name-only strings because Anthropic publishes robots.txt tokens rather than fixed full strings. Google-Extended carries no independent user agent string because Google handles those fetches using Googlebot, so our probe marks Google-Extended as Not sent. Because our probes originate from Layer3Labs test servers rather than official vendor IP ranges, edge firewalls configured to verify authentic crawler IP addresses may refuse our test request even if real search crawlers are allowed through. Treat a Refused result as a reason to check your firewall logs and bot settings. It does not prove the real crawler is blocked.

Fixing Blocked Search Crawlers

To restore access for a blocked search crawler, fix both its robots.txt rule and any firewall, CDN or hosting bot rule. If the test indicates that search crawlers are Blocked by robots.txt, use the lines in the copy box to grant access. The AI crawler checker generates explicit groups for blocked search indexers, pairing User-agent lines for OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot with an Allow: / directive. You can copy these groups directly into your robots.txt file using the Copy robots.txt Lines button.

When you add a dedicated User-agent group for a specific crawler, remember that the crawler stops reading directives under User-agent: *. Any global exclusion rules you maintain for sensitive directories, such as Disallow: /admin or Disallow: /checkout, will be ignored by that crawler unless you explicitly duplicate those Disallow lines within its dedicated block.

Changes made to your robots.txt file require time to propagate across external systems. OpenAI notes in its documentation that robots.txt updates take about 24 hours to take effect across ChatGPT crawling systems. Once your updated file is published, run the checker again to confirm the search crawlers show Can reach the page. If you are working to expand citations in conversational platforms, consult our technical GEO checklist and our breakdown on how to get cited in ChatGPT.

Limits of the Crawler Checker

The AI crawler checker measures technical crawler access and text directives on a specific URL. It does not measure search rankings, brand mentions, or live AI citations. A page can achieve perfect access across all eleven crawlers while failing to appear in conversational search answers. Google documents in its AI features guidance that to appear as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to appear in Google Search with a snippet.

Our evaluation tests one single URL per submission. Because robots.txt directives and internal firewall rules can differ by path, checking your homepage will not reveal whether a blog post, resource guide, or product directory is restricted. Check a blog post or product page as well as your homepage.

Robots.txt functions as a voluntary protocol that automated systems choose to honor. Direct user-requested fetchers like Perplexity-User or ChatGPT-User may ignore robots.txt restrictions when fulfilling direct user browsing commands. Furthermore, because our test requests originate from our test infrastructure, enterprise firewalls that validate bot IP blocks may reject our probe. If you need to monitor live brand citations within conversational answers, review our guide to the best AI visibility tools. You can also test your site's LLM context file using our free llms.txt checker.