Which AI Crawlers Are You Blocking Without Knowing?
Four AI crawlers matter most: GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity) and Google-Extended (Google AI training). Each obeys robots.txt. A single 'Disallow: /' under a wildcard user-agent — or a copied block-all-AI snippet — removes you from AI answers entirely.
The crawlers that matter
| User-agent | Operated by | What it feeds |
|---|---|---|
GPTBot | OpenAI | ChatGPT browsing and training |
ClaudeBot | Anthropic | Claude's web access |
PerplexityBot | Perplexity | Perplexity answers and citations |
Google-Extended | Gemini and AI Overviews training |
One nuance that catches people out: Google-Extended does not
control whether you appear in Google Search. Blocking it opts you out of AI
training uses while leaving normal search indexing untouched — those are
separate decisions, controlled separately.
How blocking happens by accident
Almost nobody sits down and decides to be invisible to AI search. It happens three ways:
- A copied snippet. "Block AI scrapers" robots.txt blocks circulated widely in 2023-24. Plenty were pasted in and forgotten.
- A wildcard rule.
User-agent: *followed byDisallow: /blocks every crawler, AI or otherwise. Common on staging configs that got promoted to production. - A security plugin or CDN rule. Some WAF presets block unfamiliar user-agents by default, which now includes the AI crawlers.
The failure is silent. Nothing breaks, no error appears in Search Console — you simply stop being cited, and there's no dashboard telling you why.
See this on your own site. isaeo audits up to 10 pages free — no card, no trial timer. You get an SEO health score, an AEO score and a prioritised fix list.
Run a free audit →Should you block them?
This is a real business decision, not a technical default, and the honest answer depends on how you make money.
Reasons to allow: if you sell software, services or anything with a consideration phase, being cited in an AI answer is free top-of-funnel distribution to someone actively researching. Blocking removes you from that channel while your competitors stay in it.
Reasons to block: if your content is the product — a paywalled archive, a proprietary dataset, licensed journalism — then letting models ingest it for free may undercut the business directly.
Most SaaS and agency sites fall clearly in the first camp and block by accident anyway. That's the case worth catching.
How to check
Open yourdomain.com/robots.txt and look for any
Disallow: / under a wildcard or an AI user-agent. Remember that
group selection matters: if you declare a GPTBot group, the
* rules stop applying to GPTBot entirely.
isaeo checks all four crawlers on every audit and caps your site-level AEO score if a key one is blocked — because no amount of structured data matters if the engine can't fetch the page. It also checks for llms.txt in the same pass.
Try it on a site you care about
Find out in two minutes whether you're blocking the crawlers that feed AI answers.
Start free — 10 pages, no cardNeed full-site crawls, scheduled audits and white-label reports? Compare plans →
Frequently asked questions
What is GPTBot?
GPTBot is OpenAI's web crawler. It respects robots.txt, so a Disallow rule targeting GPTBot or a wildcard user-agent prevents OpenAI from crawling your site for ChatGPT.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content is used for Gemini and AI training. Normal Google Search indexing is controlled by Googlebot and is unaffected by blocking Google-Extended.
Should I allow AI crawlers?
If your content markets a product or service, allowing them is usually right — AI citations are free distribution to people researching a purchase. If your content is itself the paid product, blocking may protect the business.