Block the Wrong Bot and You're Deleting Yourself From AI Answers

The most expensive misconception: “blocking AI” isn’t one switch

“I don’t want AI taking my content to train on” — a reasonable instinct, so many people go into robots.txt and block every AI-related crawler. The problem: a single AI company often sends more than one crawler, each doing completely different things. Block them all and you usually also cut the one that gets you cited.

The result is the most ironic kind of failure: you think you’re only refusing training, but you’re actually deleting yourself from AI’s answers — and because it’s invisible, you have no idea.

Training bots and search bots are different crawlers

Most mainstream AI companies split their crawlers into two purposes: one fetches content to train models, the other fetches in real time when a user asks a question, then cites you. Blocking the former doesn’t hurt visibility; blocking the latter means leaving that engine’s answers.

The point isn’t to memorize this table — it’s to understand that “block training” and “keep visibility” can both be true at once, as long as you can tell which bot is which.

The exception: Google-Extended is not a clean “training switch”

One cell in that table deserves its own section. According to Google’s own documentation, Google-Extended controls two things: whether your content can be used to train future Gemini models, and whether the Gemini app can use your pages to ground its answers. The same page says blocking it doesn’t affect your site’s inclusion in Google Search and isn’t used as a ranking signal. Google’s October 30 announcement puts it more plainly: sites that specifically allow Google-Extended are eligible to appear in the Gemini app’s answers, where customers can click through to them.

So if you block it, you’re out of the Gemini app’s answers by design. A large-scale measurement study (SIGIR 2026, arXiv:2604.27790, data from December 2025) found something similar. The researchers tested Gemini connected to Google Search through the API, not the phone app. Twenty-one major sites, from The New York Times, the BBC and Reuters to Nature, each appeared in the results for at least 20 queries in both Google Search and AI Overviews, yet were never once cited by Gemini; on closer inspection, all 21 block Google-Extended.

The surprise is AI Overviews. Google’s documentation says blocking it doesn’t affect Search inclusion, and AI Overviews is part of Google Search, so it shouldn’t be affected; yet the same study found that sites that block Google-Extended are measurably less likely to be cited by AI Overviews.

Nobody has a confirmed explanation for the AI Overviews half yet, but the takeaway is clear: treating Google-Extended as a “low impact if blocked” pure training switch takes you straight out of the Gemini app and gambles with your AIO visibility. If you care about AI visibility at all, leave Google-Extended allowed — and if you genuinely want out of training, start with the other vendors’ training UAs.

The cost of getting it wrong: vanishing from AI answers

In traditional SEO, blocking the wrong crawler at least shows up as a ranking drop you can notice. But AI citation is invisible: a user asks, AI answers, you’re not in it — no notification, no “impression that didn’t happen” in any dashboard.

That’s what makes blocking the wrong AI bot so dangerous: there’s no alarm. By the time you notice “competitors get mentioned in ChatGPT and I don’t,” you’ve usually been missing out for a long time.

noai and robots.txt are not the same thing

A common confusion worth clearing up: the noai / noimageai meta tags on a page, and robots.txt crawler rules, are two different mechanisms. The former asks “don’t train on this page”; the latter controls “which crawler may fetch which paths.” Both rely on crawlers honoring them voluntarily, neither is an enforceable standard, and both can hurt your visibility if set too bluntly.

So how should you set it

In one line: block training, allow search.

• To opt out of training, write rules for the training UAs (GPTBot, ClaudeBot, Applebot-Extended…) — Google-Extended is the exception: blocking it takes you out of Gemini app answers, and the study above found such sites get cited less often in AI Overviews;
• Always allow the search / live-citation UAs (OAI-SearchBot, Claude-User, Perplexity-User…), or you’re actively opting out of AI answers;
• After editing, cross-check against each vendor’s crawler docs to confirm you blocked the bot you think you did.

For each vendor’s full crawler list and rule differences, see the earlier post: The 8 major AI crawlers — rule differences and best settings.

Why this isn’t “set it once and forget”

AI companies add and rename crawlers (it’s changed several times in the past two years), and one typo in robots.txt — or one default toggle in Cloudflare — can shut your whole site to a given bot. Combined with the fact that lost AI citations come with no alarm, this isn’t a set-once task: it’s ongoing site-health maintenance that requires cross-checking the latest crawler lists and verifying regularly — exactly the kind of invisible, slow-bleed problem that’s best watched continuously rather than discovered after the damage is done.