What happened
On July 1, 2026, Cloudflare pushed its AI-crawler policy a step further. Blocking crawlers used to be something you set up yourself in robots.txt; now, for newly onboarded domains on the free plan, Cloudflare blocks them by default — and for the first time it splits crawlers into three kinds, handled separately: search (fetches your content to index it and answer questions about it later), agent (acts in real time on a user’s behalf to get something done), and training (scrapes your content to train models). What’s blocked by default is the latter two, and only on pages that display ads — search crawlers stay allowed, and site owners can switch the default off in the dashboard.
It also drew AI companies a line: by September 15, separate the identities of your “search” and “training” crawlers, or the ones that blur the two get blocked. Behind this is the paid infrastructure Cloudflare has been laying down all year — Pay Per Crawl (charging crawlers per fetch via HTTP 402) and Monetization Gateway — turning “AI scrapes your content for free” into “want to crawl? name a price first.”
One thing the headlines overstated: you’ll see claims of “100% block, no exceptions after 9/15.” That’s inflated. Per Cloudflare’s own announcement, the default block covers training plus agent, only on ad-displaying pages, mainly for new and free-tier customers — and it leaves an opt-out. It moves the default; it doesn’t weld the door shut.
Why it matters to you
The first instinct is: “Great, now AI has to pay to use my content.” For the vast majority of brands, this has nothing to do with getting paid — the ones collecting crawl fees are Reddit and the large publishers; your site more likely won’t even get through the door of a paid marketplace.
More worth reading than the fees is why Cloudflare is doing this. The numbers it laid out: AI crawlers scrape enormous amounts of content and send almost no traffic back. Cloudflare CEO Matthew Prince has said in several interviews that OpenAI’s crawler fetches well over a thousand pages for every 1 visitor it sends back, while traditional Google search returns one for every dozen or so. A Pew Research Center study from July 2025 found that when a Google page shows an AI summary, the share of users who click a traditional search result link below drops from 15% to 8% — nearly halved. Scraping up, clicks gone — even the infrastructure layer has had enough, and is now blocking AI by default.
That “default block” carries one concrete risk for you: if your site sits behind Cloudflare — or your CDN follows with the same default — you could land on the “blocked” side without knowing. And blocking is guilt by association: what gets cut isn’t only the training crawler, but potentially the search crawlers that would fetch you in real time and cite you in an answer. Disappearing from AI answers comes with no alarm; by the time you notice a competitor getting mentioned in ChatGPT and you don’t, a lot of time has usually passed. (For how to set that line just right, Don’t block the wrong bot goes deeper.)
What to do now
Don’t do two things because of this news: don’t rush to celebrate “finally, I can charge AI” (most can’t collect), and don’t follow the crowd in blocking every AI crawler (you’ll likely block yourself too).
There’s really one small thing and one big thing. Small: if you use Cloudflare, check the dashboard to see whether this new default swept up the search crawlers you meant to keep. Big: put your energy back into being the one who still gets cited no matter how the platform changes its defaults. Who’s blocked, who’s allowed, whether crawling costs money — all of it has flipped several times in two years and will flip again; the only thing that survives every flip is the set of conditions that make AI willing to name you in its answer. That’s the ground you need to stand on, well before any question of crawl-fee revenue.
robots.txt templates are all over the internet; the hard part was never pasting one in — it’s watching, in an environment where the default gets quietly switched under you with no alarm, which crawler got blocked and which citation dropped. That’s a job someone has to tend, not a set-and-forget.