Which AI crawlers can see your site?
The crawler that trains a model is not the crawler that finds you. A business can allow every AI-branded bot it has heard of and still be invisible when a customer asks, because the bots that decide are the ones that do not look like AI crawlers at all.
Which crawler decides whether each assistant can see you
| Where you want to be named | The crawler that decides it | What the vendor says happens if you block it |
|---|---|---|
| ChatGPT search | OAI-SearchBot | “Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.” |
| Claude search | Claude-SearchBot | Blocking it “prevents our system from indexing your content for search optimization, which may reduce your site’s visibility”. |
| Claude, answering a question | Claude-User | Blocking it “prevents our system from retrieving your content in response to a user query”. |
| Google AI Overviews | Googlebot | Blocking it removes you from Google Search and from AI Overviews together. |
| Brave Search | no user agent of its own | “If a domain or page is not crawlable by Googlebot, then Brave Search’s bot will not crawl it either.” |
| Perplexity | PerplexityBot | Perplexity maintains its own index, separate from Google’s and Bing’s. |
| Gemini training | Google-Extended | Controls Gemini and AI training only — not Google Search ranking. |
| OpenAI training | GPTBot | Training only. Allowing it does nothing for ChatGPT search visibility. |
| Anthropic training | ClaudeBot | Training only. Allowing it does nothing for Claude search visibility. |
Every row above is taken from the company’s own crawler documentation, read on 7 October 2026: OpenAI, Anthropic and Brave.
The training bot is not the search bot.
GPTBot and ClaudeBot are the names everyone knows, and both control training. Neither decides whether an assistant can find your business today. That is OAI-SearchBot, Claude-SearchBot and Googlebot — and not one of them reads like an AI crawler at a glance.
Brave is the sharpest case, because there is nothing in your robots.txt to look at. Brave’s crawler ships no user agent of its own and follows Googlebot’s rules, so a single blocked line removes you from a search engine your file never mentions.
We checked the claims, including our own.
“ChatGPT answers from Bing’s index, so you need Bingbot.” OpenAI documents OAI-SearchBot as the control for ChatGPT search. Bingbot governs Bing and Microsoft Copilot.
“Claude answers from Brave’s index.” Anthropic runs Claude-SearchBot and indexes for itself.
Both claims are repeated widely, and both were in our own tooling until we checked them against the vendors’ pages and corrected it. We mention that because it is the actual lesson: a claim everyone makes is not a source.
What owners ask about this.
Which crawler decides whether ChatGPT can see my business?
OAI-SearchBot. OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. GPTBot is a separate crawler and controls training only, so allowing GPTBot does nothing for whether ChatGPT can find you.
Does allowing ClaudeBot make my site visible in Claude?
No. Anthropic runs three robots and ClaudeBot is the training one. Claude-SearchBot is what indexes your content for Claude's search, and Claude-User is what fetches your page when someone asks Claude a question. Allowing only ClaudeBot leaves both of those blocked if your robots.txt blocks them.
Does blocking Googlebot affect anything besides Google?
Yes. Brave Search's crawler does not advertise a user agent of its own, and Brave states that if a domain or page is not crawlable by Googlebot then Brave Search's bot will not crawl it either. So one blocked line costs you Google, AI Overviews and Brave together, and no line in your robots.txt mentions Brave at all.
Is robots.txt the only thing that can block an AI crawler?
No. Some hosts and content delivery networks block AI crawlers independently of robots.txt, so a permissive robots.txt does not by itself prove a crawler can reach you. Brave also notes that robots.txt is not what keeps a page out of an index, a robots noindex directive is what delists a page.
By Steven Allbright, founder, Ultimate Elite Solutions · Last updated: 2026-10-07