TL;DR

Not all AI crawlers are the same. Training-data crawlers (GPTBot / ClaudeBot / Google-Extended / CCBot / cohere-ai) and live-citation crawlers (ChatGPT-User / PerplexityBot) can have their permissions configured separately. This article breaks down the differences between the 8 major crawlers and provides a practical robots.txt you can use.

Why manage them separately?

Many people treat “opening up to AI” as one undifferentiated thing, but the two kinds of crawler have completely different intents:

In theory you should prioritize live citation and be selective about training. In practice most sites simply open everything up or block everything, but configuring the two separately is the smarter strategy.

The 8 major AI crawlers — configure the two purposes separately Training-corpus crawlers Live-citation crawlers Take content to train the model; tacit model knowledge Fetch in real time on a user query — direct GEO citation GPTBot ClaudeBot anthropic-ai Google-Extended CCBot cohere-ai ChatGPT-User PerplexityBot OpenAI training Anthropic training Anthropic general Gemini training / grounding Common Crawl open data Cohere training ChatGPT browse Perplexity live Recommendation: strongly advise allowing → directly decides whether the AI recommendation pool can include your site Block it = disappear from AI recommendations for good

The 8 major AI crawlers at a glance

User-Agent Company / Product Purpose Honors robots.txt Recommended setting
GPTBot OpenAI Trains GPT models Yes Allow (unless you have copyright concerns)
ChatGPT-User OpenAI ChatGPT browse / live citation Yes Allow
ClaudeBot Anthropic Trains Claude models Yes Allow
anthropic-ai Anthropic Legacy / general Yes Allow
PerplexityBot Perplexity AI Search + live citation Yes Allow
Google-Extended Google Trains Gemini models + grounding in Gemini Apps / Vertex AI Yes Allow
CCBot Common Crawl Open dataset (used by many LLMs as a training source) Yes Allow
cohere-ai Cohere Trains Cohere models Yes Allow

Note: Googlebot and Bingbot are traditional search-engine crawlers and aren’t on the AI-crawler list, but Google AI Overviews citations run on the Googlebot + snippet-control directives lane; Google-Extended only governs Gemini training and grounding and is not an AI Overviews switch (see the snippet control deep dive) — blocking Googlebot means blocking traditional search and AI Overviews at once.

Paste the following into https://yoursite.com/robots.txt:

# Open up to all AI crawlers, but protect internal endpoints like /api/
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/

# Major AI / LLM crawlers — allow explicitly (so a site-wide *=disallow doesn't accidentally block them)
User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /api/

User-agent: anthropic-ai
Allow: /
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: Google-Extended
Allow: /
Disallow: /api/

User-agent: CCBot
Allow: /
Disallow: /api/

User-agent: cohere-ai
Allow: /
Disallow: /api/

Sitemap: https://yoursite.com/sitemap.xml

Content-Signal: finer-grained permission control

The IETF’s recent draft-ietf-aipref-content-signal proposal lets you configure three purposes separately:

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

What it means: - search=yes — allow traditional search indexing - ai-input=yes — allow live AI citation (ChatGPT browse, Perplexity, AI Overviews) - ai-train=no — do not allow use as training data

geoweb.tw configures it exactly this way — happy to be cited, but reserving training rights. Support across the various AI vendors is still evolving, but this is the future.

Counter-examples: some common misconfigurations

❌ Blocking everything

User-agent: *
Disallow: /

For GEO, this means “I never want to be cited by AI.” Even if your content is worth citing, if AI can’t reach it, you get 0.

❌ Blocking AI training but forgetting live citation

User-agent: GPTBot
Disallow: /

But this misses ChatGPT-User — when a user asks a question in ChatGPT, the agent still crawls your site, yet you forgo the long-term value of contributing to the training corpus. The worst of both worlds.

❌ Disallow, but serving AI an empty shell of content

Some sites return an empty <html></html> page to AI crawler UAs to try to game the system — AI vendors can already detect this kind of “cloaking,” and once caught, they’ll permanently flag your site as “low quality.” Be honest.

How to verify your settings are working

  1. Run curl -A "GPTBot/1.0" https://yoursite.com/ and check the response is the same as a normal browser
  2. Check that https://yoursite.com/robots.txt has the correct content
  3. Run the free GEO health check — the 8th dimension, “AI crawler reachability,” shows you bot by bot which ones are allowed

Further reading