Is a firewall hiding you from AI crawlers?
We request your page as an ordinary browser, as an unknown robot and as each AI crawler — then compare the three. Your robots.txt can say yes while your edge says no, and nothing warns you.
Takes a few seconds — the requests go one at a time on purpose. We do not store the pages people check.
Anyone who has fixed everything and is still not named.
You wrote the pages. The schema validates, the sitemap resolves, robots.txt allows everything, and an assistant asked who to hire in your category still does not mention you.
There is a reason that no content audit can find: the crawler may never be getting a response at all. Not refused by your robots.txt, refused by whatever sits in front of your site, before your website is even consulted.
It is the one failure where everything you can see is correct.
Six things, and two of them are about whether we can trust the answer.
A flat refusal
We ask for your page as each of 4 AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot — and report a 401, 403, 406 or 451 for what it is. That is the crawler being turned away before it reads a single word.
A browser challenge
Worse than a refusal, because it looks like a page. A challenge only clears if the client runs JavaScript and keeps a cookie. No crawler does either, so a challenge is a block that reports itself as a success.
A throttle instead of a block
Some sites rate-limit a crawler rather than refuse it. We retry once, spaced, and only call it a throttle if everything else in the same run was served, otherwise the limit was probably ours.
Which product is in front
Cloudflare, Akamai, Fastly, CloudFront, DataDome, HUMAN, Imperva, Sucuri. We name the one making the decision rather than the one delivering the bytes, because they are frequently different and only one has the setting you need.
Whether the rule targets AI at all
The question everything else skips. A refused crawler proves nothing on its own, a site that refuses all unfamiliar robots refuses ours too. The control request is what separates the two.
Whether we could tell at all
If your site will not serve our ordinary browser request, there is nothing to compare against and we say so. An inconclusive answer is reported as inconclusive, never as clean.
A request is not enforcement.
robots.txt is a published request. It is a file on your site saying which crawlers you would like to allow, and a well-behaved crawler reads it and complies voluntarily. It has no power to stop anything.
A firewall or bot-management rule is enforcement. It decides whether a request gets a response. It runs before your website does, so a crawler refused there never reaches your robots.txt, and the permission you wrote in it is never read by anyone.
This is why a robots.txt checker will tell you a comprehensively blocked site is clean. Ours will. It reads the request and reports it accurately, and the request is being honoured by every crawler that is allowed to ask. The ones being turned away at the door never got to read it.
The reason this matters now is that the blocking usually was not a decision. Cloudflare has shipped AI-bot blocking enabled by default for new zones, and several hosting and security plans do something comparable. It arrives with the service. Nobody sends an email about it.
Three kinds of request, because one proves nothing.
If we only asked as GPTBot and got a 403, we would know almost nothing. We are not arriving from OpenAI's network, so a site that refuses every unverified robot refuses us too. And reporting that as “this site blocks AI” would be a false alarm on nearly every well-defended site on the internet.
So each run makes three kinds of request to the same address:
| Request | What it establishes |
|---|---|
| An ordinary browser | Whether this site will serve us at all. If not, nothing else in the run means anything and we say the result is inconclusive. |
| An unknown robot (the control) | Whether any unfamiliar automated client gets refused here. This is the request that separates a real finding from a false one. |
| Each AI crawler | Whether the rule keys on the crawler's name specifically. |
Read the three together and the answer is unambiguous. Browser served, control served, GPTBot refused. The rule is about the crawler. Browser served, control refused, GPTBot refused, the site refuses robots it does not recognise, which is a different fact and we report it as that one.
Every request carries the crawler's token and our own name and address. The token has to be there or the rule never fires; our name is there so anyone reading their access log can see precisely who called.
The honest limits.
We are not the real crawler
We send the crawler's name from our own server, not from OpenAI's or Anthropic's network. A rule that verifies a crawler by its network can treat us differently from the real one, in either direction. This is strong evidence, not proof.
One page, one moment
We check the address you give us, once. A rule can apply to one path and not another, and bot scoring changes with traffic. A clean result on your homepage does not clear your whole site.
We cannot see your rules
We can tell you a crawler was refused and which product answered. We cannot read your configuration, so we cannot tell you which rule did it or switch it off.
Not a robots.txt check
This is the other half. robots.txt is what you ASK crawlers to do; this is what your edge actually DOES. Run the robots.txt checker as well. And note that Google-Extended and Applebot-Extended are robots.txt permissions rather than robots, so only that tool can answer those two.
Five moments worth ten seconds.
When you are invisible and nothing explains why
Pages indexed, robots.txt clean, schema valid, and no assistant ever names you. This is the check that finds the reason the others cannot see.
After switching on a CDN or security plan
The commonest cause by far. AI-bot blocking is a default on some plans, so the day you moved your DNS is the day it started, and nothing told you.
After anyone enables a “bot fight” setting
Those settings are written to stop scrapers and they catch AI crawlers in the same net. The person who turned it on was not thinking about ChatGPT.
Before paying for AI visibility work
Content and schema cannot help a crawler that never gets a response. Check the door opens before paying anyone to decorate the room.
On a competitor who gets named when you do not
Their edge configuration answers our requests exactly as it answers a real crawler's. It is a fair question to ask.
Almost everything else reads the file you wrote.
A robots.txt checker reads your own document back to you. That is useful — we publish one — and it is a check on your intent, not on your site's behaviour. Every tool in that category shares the blind spot, including ours.
This one makes the request and reads what comes back. It is the difference between reading a sign on a door and trying the handle.
And it is built to be wrong in the safe direction. A rate limit is never reported as a block, because we make several requests in a row and the limit is probably ours. A refusal is never called an AI block unless an unknown robot was served in the same run. A site that will not serve our browser is reported as inconclusive rather than guessed at. We would rather tell you we could not establish something than hand you a finding you act on and cannot reproduce.
We know which way that error runs because we built it the other way first. Pointed at five real sites, this tool named a CDN instead of the bot manager actually refusing the request, and left a throttled crawler as no answer at all. Both are fixed. Neither was visible in a test we wrote ourselves.
An open door is the floor. Being named is the job.
An unblocked crawler can reach your pages. Whether an assistant then names you when a customer asks who to hire depends on what those pages answer and whether your facts agree across every source an engine can reach. This tool clears a prerequisite, not the goal.
We publish how we test that, including the questions where we are not named, on our methodology page. Prices are on the pricing page.
What owners ask about this.
My robots.txt allows everything. Why would anything be blocked?
Because they are two different systems and only one of them is enforcement. robots.txt is a published request that well-behaved crawlers choose to honour; a firewall or bot-management rule decides whether the request gets a response at all. A crawler that is refused at the edge never reads your robots.txt, so the permission in it is never consulted. This is why a robots.txt checker will call a comprehensively blocked site clean.
Who would block AI crawlers on purpose?
Plenty of people, for good reasons — publishers protecting work they sell. But most sites we find blocked did not choose it. Cloudflare has shipped AI-bot blocking enabled by default for new zones, and several hosting and security plans do something similar, so the setting arrives with the service rather than from a decision. If you sell a service and want to be recommended, it is working against you.
What is the control request, and why does it matter?
It is an ordinary request from an unremarkable robot that names no AI company. Its only job is to answer one question: would ANY unfamiliar automated client have been refused here? If the control is served and the AI crawlers are not, the rule keys on the crawler's name and that is a real finding. If the control is refused too, the site simply refuses robots it does not recognise, and we say that instead. Without this request the tool would report an AI block on nearly every well-defended site on the internet.
You send a user agent saying GPTBot. Is that dishonest?
The agent has to contain the crawler's name or the rule being tested never fires, and the tool would report all clear on a site that blocks everything. So every request we send carries the crawler's token AND our own name and address alongside it. Anyone reading their access log sees exactly who called and can look us up. We are diagnosing, not sneaking in.
The result says inconclusive. What does that mean?
That your site did not serve our ordinary browser request either, so there was nothing to compare an AI crawler against. It is neither a clean result nor a block. It means the answer has to come from your own edge logs. We would rather tell you we could not establish something than guess. Large publishers behind aggressive bot management land here routinely.
A crawler came back throttled rather than blocked. Is that bad?
It is milder and it is still a problem. A crawler slowed to a trickle indexes a fraction of your site, which reaches the same place more slowly. We only call it a throttle when the limit survived a spaced retry while a browser, our control robot and other AI crawlers were all served normally in the same run, if everything was limited, the cause was our own request rate and we say so instead.
How do I fix a block once I find one?
It is a setting, and it is usually one switch. In Cloudflare it lives under Security, look for AI Scrapers and Crawlers, plus any bot-fight mode and any custom rule mentioning a crawler by name. Other products put it under bot management. Decide deliberately which crawlers you want: the training crawlers and the ones that answer a user's question right now are different robots, and blocking the first does not have to mean blocking the second.
Do you store the pages people check?
No. The check runs, the result goes to your browser, and nothing is kept. There is no sign-up and no email field on this page.
Related: Robots.txt checker · Sitemap validator · Structured data validator · Canonical tag checker · Title tag checker · llms.txt generator
By Steven Allbright, founder, Ultimate Elite Solutions · Last updated: 2026-10-09
The next checks worth running.
Each one takes seconds and needs no sign-up. They fail in order, so start at the top.
Robots.txt checker · Sitemap validator · Structured data validator · How we run a real test · Get the free report