Which AI crawlers can read your site?
Enter a domain. We read its robots.txt and tell you, crawler by crawler, what is allowed, what is blocked, and which line is doing the blocking.
We do not store the domains people check.
Anyone who expects to be recommended by an assistant.
Run it if you have just moved host, changed CDN, installed a security plugin, had a developer “harden” the site, or inherited a website whose history nobody can account for. Those four are how almost every block we find arrived.
Run it also if you are paying anybody for AI visibility work. Crawler access is the floor underneath everything else that gets sold on top of it, and it takes ten seconds to confirm. If the floor is missing, nothing built on it can work. And a report that never mentioned it is a report worth questioning.
It is not for diagnosing why you rank below a competitor. That is a different question with a longer answer, and this tool deliberately does not pretend to answer it.
A name you recognise is the wrong thing to look for.
The column that matters is the middle one: what that crawler’s access decides. Some of the best-known AI crawler names control model training and have no effect on whether an assistant can find you today, while two crawlers that do not look like AI at all decide exactly that.
So read down the middle column, not the first one. If a row that decides search visibility says Blocked, the line underneath it is the line to change in your file.
When a crawler is blocked, the tool prints the line that did it and the User-agent group it sat under. That is deliberately the whole fix: you can search your own file for that exact text.
We wrote the full explanation, with every claim traced to the vendor’s own documentation, on a separate page: which AI crawlers can see my site.
18 crawlers, in three groups that matter differently.
Most checkers hand you a list of names and leave you to work out which ones matter. They do not matter equally, and the grouping below is the point of the tool.
Crawlers that decide whether an assistant can find you
Block one of these and you are not in that engine’s answers at all. This is the group that costs you customers, and it is the group almost nobody checks.
OAI-SearchBot
ChatGPT search resultsClaude-SearchBot
Claude’s search index — blocking it removes you from Claude’s search resultsGooglebot
Google Search and AI Overviews — and Brave Search, whose crawler follows Googlebot’s rulesBingbot
Bing search and Microsoft CopilotPerplexityBot
Perplexity’s indexDuckAssistBot
DuckDuckGo’s AI answersAmazonbot
Alexa and Amazon’s assistants
Crawlers that open one page to answer one question
These fetch a specific page when a user asks the assistant to look at it. Blocking them does not remove you from an index, but it does mean the assistant cannot read your page while it is answering.
ChatGPT-User
pages ChatGPT opens when a user asks it to browseClaude-User
pages Claude opens in answer to a user’s questionPerplexity-User
pages Perplexity opens for a specific answer
Crawlers that collect text to train models
These are the famous names, and they are the ones businesses are usually told to allow. Allowing or blocking them changes nothing about whether a customer is told your name today. It is a decision about future models, which is a real decision, just not this one.
GPTBot
OpenAI’s model trainingClaudeBot
Anthropic’s model training — NOT whether Claude can find you in searchanthropic-ai
Anthropic, older token still used by some sitesGoogle-Extended
Gemini and Google’s AI training — NOT Google Search rankingApplebot-Extended
Apple IntelligenceCCBot
Common Crawl, which many models are trained onmeta-externalagent
Meta AIBytespider
ByteDance and TikTok search
What each crawler feeds is taken from that company’s own documentation. The grouping is ours.
Four steps, and nothing is kept.
1. You enter a domain. We request /robots.txt at that address, following redirects, a bare domain almost always redirects to www, and reading that redirect as “no file” is how checkers report a clean result for a site that blocks everything.
2. We parse the file into User-agent groups and their rules, the way a crawler does: a named group wins over *, and the rules that follow a run of agent lines belong to all of them.
3. Each of the 18 crawlers is matched against those groups, and we record not just the verdict but the exact line that produced it.
4. We add the consequences that no row can show, an engine whose crawler advertises no user agent of its own still depends on one that is listed, and that dependency is invisible in your file.
The answer returns to your browser. The domain is not logged, and there is nothing to unsubscribe from.
The honest limits, because the gaps matter.
Your CDN or firewall
robots.txt is a request, not a lock. Cloudflare can block AI crawlers by default whatever your file says, and this tool cannot see that. It is the likeliest reason a site with a perfect robots.txt is still invisible.
Page-level blocks
A noindex meta tag or an X-Robots-Tag header can remove a single page that your robots.txt happily allows.
Whether your content can be read
A crawler that is allowed in, but finds a page that renders only after JavaScript runs, may come away with nothing.
Whether you are actually cited
The most important gap. Access means the door is open. It says nothing about whether an assistant names you when a customer asks. That depends on what is on the page, and what the rest of the internet says about you.
We publish the raw output of our own citation testing, including the answers that did not name us, at our methodology page.
Three things to check, in this order.
Your robots.txt
The tool hands you the exact line and the User-agent group it sat under. Most blocks are inherited from a bad-bot list someone pasted in years ago, not a decision anybody made.
Your CDN or firewall
A file can allow a crawler that your edge then refuses. Cloudflare blocks AI crawlers by default unless that setting is turned off, and nothing in robots.txt will tell you.
Your pages
An open door is not an answer. A crawler that can reach a page which never answers the customer’s question still has nothing to cite. The first two are an afternoon. This one is the work.
Most tools answer a question nobody asked.
A validator tells you the file parses. That is a question about syntax. A perfectly valid robots.txt can block every engine that would have sent you a customer.
A generic crawler list tells you which bots are mentioned. Being mentioned is not the same as being blocked, and not being mentioned is not the same as being allowed, a User-agent: * group governs every crawler that has no group of its own, which is most of them on most sites.
An AI-bot checker looks for AI-branded names. It will therefore miss the two plain search crawlers that decide whether an answer engine can see you, and it cannot show a dependency for an engine that ships no crawler name at all.
This one reports what each crawler’s access decides, prints the line responsible, and names consequences for engines that have no row in any table. We also publish our own site’s result rather than only other people’s. And ours is not spotless.
Five moments where this has actually found something.
After a migration or redesign
A staging site’s Disallow: / going live with the rest of the build is the classic one, and it blocks everything.
After a new plugin
Several security and SEO plugins write their own rules, and several ship a bad-bot blocklist that has had AI crawlers added to it over the years.
When an agency reports no AI visibility
Before paying for a strategy, check whether the door is shut. Ten seconds, yes or no.
Before publishing for AI search
Writing answer pages for engines that cannot fetch them is the most expensive mistake available in this field.
On competitors who keep getting named
Their file is public. Sometimes the whole explanation is that they are readable and you are not.
Access is the floor. Being named is the job.
This tool answers one question honestly and then stops. The work we actually do starts after it: finding the questions your customers ask an assistant, answering them on pages an engine can verify, and establishing the facts about your business — name, address, registration, prices — consistently enough that an engine will repeat them.
We test that the way we would want a supplier to test it: by asking the assistants, recording what they answer, and publishing the results including the ones where we are not named. That is our methodology, and the raw answers are on it.
Prices are published on the pricing page. There is no setup fee and no sales call required to see them.
A blocked crawler is a silent failure.
Nothing tells you. There is no bounce, no error, no drop in a dashboard you already read. Your site stays up, your traffic report looks ordinary, and an assistant simply never mentions you to the person asking who to hire.
That is what makes it different from most marketing problems. A bad advert performs badly and you can see it. A blocked crawler produces no signal at all, the opportunity never arrives, so there is nothing to measure and nothing to react to.
And the fix, when there is one, is usually a single line in a text file. The cost is almost entirely in not knowing.
What owners ask about this.
What does this actually check?
It reads the robots.txt at the domain you enter and works out, crawler by crawler, whether that file allows or blocks it. And when something is blocked, which line did it. It checks 18 crawlers, including the two plain search crawlers that decide whether an answer engine can see you.
Why does it say my site is fine when I have no robots.txt?
Because that is correct. No robots.txt means no restrictions, and every crawler is allowed. A missing file is not a problem to fix; it is the most permissive possible state.
I allow GPTBot and ClaudeBot. Is that enough?
No. Those two control model training. Whether an assistant can find your business when a customer asks is decided by different crawlers, and allowing the famous ones buys you nothing in search. That is the single most common mistake we see, and it is why this tool reports what each crawler's access decides rather than just listing names.
The check found nothing blocked, but we still are not named by AI. Why?
Crawler access is necessary, not sufficient. It only means the door is open. Being named also needs the page to answer the question a customer actually asks, in the words they ask it, with facts an engine can verify. Access is the floor, not the product.
Does a blocked crawler always mean someone blocked it on purpose?
Rarely. Most blocks arrive by inheritance, a bad-bot blocklist pasted in years ago, a security plugin's default, or a CDN setting nobody chose. Our own robots.txt is 24KB and close to 700 disallow rules from exactly that history, which is why we wrote this instead of assuming.
Do you store the domains people check?
No. The check runs, the answer comes back to your browser, and nothing about it is kept. There is no sign-up and no email field on this page.
Can I check a competitor's site?
Yes. A robots.txt is a public file at a fixed address on every website, so this reads nothing a browser could not. It is a reasonable thing to run on the firms you compete with, and the result often explains why one of you gets named and the other does not.
Why does your tool check search crawlers like Googlebot? They are not AI.
Because they decide AI answers anyway. Google's AI Overviews answer from Google's index, and Brave Search's crawler advertises no user agent of its own. It follows Googlebot's rules, which means Googlebot's access also governs an engine whose name appears nowhere in your file. A checker that only looks for AI-branded crawler names will miss both.
I blocked these on purpose. Should I unblock them?
Not necessarily, and we are not going to tell you to. Publishers block AI crawlers deliberately, because their business is selling the content those models would absorb. If your business is being hired by local customers, the calculation is the opposite: you want to be found. The tool reports what your file does; whether that is right for you is your decision.
The result says the check was inconclusive. What does that mean?
That we could not establish what your file says, usually a redirect we would not follow, or a server that answered with an error. It is deliberately not reported as 'nothing is blocked', because those are different answers and only one of them is safe to act on. Try the exact address your site serves, normally the one with www.
How often should I run this?
After any change to hosting, your CDN, your security plugin, or your robots.txt. And once a quarter otherwise. Blocks are almost always introduced by something else, which is exactly why nobody notices them.
Related: Which AI crawlers can see my site · How we test AI answers · AI visibility service · Website visitor identification · Pricing
By Steven Allbright, founder, Ultimate Elite Solutions · Last updated: 2026-10-08
The next checks worth running.
Each one takes seconds and needs no sign-up. They fail in order, so start at the top.
Sitemap validator · Structured data validator · Canonical tag checker · How we run a real test · Get the free report