On October 30, Google, OpenAI and Microsoft all published posts saying they will support a marker that websites can add, and adding it amounts to telling them, “don’t use my content to train AI.” The wording comes from a draft submitted by an employee of the AI company Anthropic, who also said Anthropic has already started work on it. Posts are already going around claiming that one line on your website is enough to make the four big AI companies comply at once.

Our advice, though, if you want AI to recommend your business, is to hold off on adding it. The way Google plans to read it, the marker may end up blocking more than training: the chance for the Gemini app to cite your website when it answers customers could go too.

What it is

The marker goes in robots.txt, a small file on your website written for the programs that AI companies and search engines send out to collect pages, the ones usually called crawlers. In practice the marker is two lines:

User-Agent: __usage_ai-training
Disallow: /

In plain terms, nothing on the site should be used to train AI. It comes from a draft that Anthropic’s Kevin Kelley submitted to the IETF in his own name on October 30. The IETF is the organization that writes internet standards, but this is only a first draft, and it has a long way to go before it becomes a standard.

The companies plan to start at different times. OpenAI says in its announcement that it will honor the marker and aims to have it working in the coming months. Microsoft, writing on the Bing blog, has set the date at February 1, 2027. Google says it plans to support it, aiming for 2027. As for Anthropic, Kelley said on the IETF mailing list that the company is already working on it, but he gave no date.

Why it matters to you

Training is when an AI company uses your pages as teaching material for its next model. Citing you in an answer is when, at the moment a customer asks a question, the AI reads your page and puts what it found, along with a link, into its reply. For a business that wants AI to recommend it, it’s being cited in answers that has a chance of bringing customers to your door.

At OpenAI, those two are set separately, and OpenAI’s own announcement says this marker is aimed at training. Google is a different story. Since 2023 Google has offered websites a control called Google-Extended, also written in robots.txt, and it covers more than Gemini training: it also decides whether the Gemini app can use your pages when it answers questions. The Gemini app is Google’s own AI chat app, where customers ask questions directly on their phone or computer.

In its October 30 announcement, Google wrote that if your site uses the marker to opt out of training and has no separate rule for Google-Extended, it plans to treat that as if you had blocked Google-Extended as well. The same announcement notes that sites which specifically allow Google-Extended are eligible to appear in the Gemini app’s answers, where customers can click through to them.

Picture an interior design firm with more than a hundred renovation articles on its website. Its biggest worry is AI learning from those articles, so after seeing one of those posts, it may well ask its web developer to add these two lines. Once Google starts supporting the marker, the chance of a link to the firm’s website showing up when customers ask the Gemini app about renovations could quietly disappear.

The AI Overview at the top of Google Search isn’t mentioned in Google’s announcement. According to Google’s own documentation, Google-Extended doesn’t affect whether a site is included in Search. But a study published this year found that sites blocking Google-Extended were significantly less likely to be picked up by AI Overviews. That is a pattern the researchers observed, and the reason isn’t clear yet. So whether this marker will also affect AI Overviews is still an open question. Opting out of AI Overviews is done separately, in Search Console, the dashboard Google gives website owners.

What to do about it

If your robots.txt doesn’t already contain this marker, you don’t need to change anything; if you want AI to recommend you, leave this marker out for now.

If you really do care about your content being used for training, be clear about one thing first: nothing Google has published so far gives you a switch that separates training from being cited in the Gemini app. Google’s announcement does leave one way in. You can add the marker and also explicitly allow Google-Extended, and your site stays eligible to appear in the Gemini app’s answers. By Google’s own definition of that control, though, you would then be allowing Gemini training as well, so in effect you would only be blocking the other companies.

The effects of this marker also won’t show up right away. On the day you add it, most AI companies won’t be acting on it yet, and the effects only appear as each one catches up; for Google, that means 2027. By then, even if your reports show fewer visitors coming from the Gemini app, it will be hard to tell whether this marker is the reason.

So whether to block or allow, you have to go through each company’s approach one by one to know what it will cost you. You can also hand this to our managed GEO service. What we look after is whether AI can find you when customers ask it, and whether it gets you right when it talks about you; settings on your website such as robots.txt are part of what we fix. Once each company starts following the marker, we go back and check that your website is still being cited.

Web developers and website platforms sometimes set up rules in your robots.txt for you, and you won’t always know. To check whether your site is blocking AI crawlers it shouldn’t, set up a time with one of our consultants and go over it in a demo. The consultant will walk you through your site’s GEO health check, which is free and includes a check on whether your robots.txt is blocking AI crawlers, and then open the Gemini app and ChatGPT on the spot to see whose websites they cite when answering questions in your line of business, and whether yours is among them.

Further reading