The question this layer is asking
When AI takes your content to train on and to generate answers, can you set conditions — or even charge? Since 2025 a whole layer of standards has competed to answer this. The main players:
| Approach | Role | Status |
|---|---|---|
| RSL 1.0 | License terms embedded in robots.txt + pay-per-crawl/inference | Self-declared standard, 1500+ brands; ~zero AI compliance |
| Cloudflare Content Signals | robots.txt flags for search/ai-input/ai-train use |
Product, 3.8M domains; advisory (voluntary) |
| Cloudflare Pay-Per-Crawl | Charge AI crawlers per request via HTTP 402 | Product, partial beta |
| TollBit / ProRata | Two-sided paid marketplaces with real money flowing | Production, still small |
| IAB CoMP | A commercial agreement before crawling | 2026/3 draft, not finalized |
The reality: many approaches, little compliance
The problem here isn’t a lack of options — it’s that they coexist, compete, and AI-side compliance is near zero. RSL calls itself a standard but AI vendors ignore it; Content Signals is advisory; the real money flows through marketplaces like TollBit and ProRata, but at modest scale. None is a finalized, enforceable standard.
The one player that actually moved the default: Cloudflare
Most of the table above is still stuck at “self-declared standard, ignored by AI.” Only Cloudflare has actually moved the default. On July 1, 2026 it split AI crawlers into search, agent, and training, and for newly onboarded free-plan domains it blocks the training and agent types by default (only on ad-displaying pages, and you can turn it off in the dashboard) — plus it gave AI companies a September 15 deadline to separate their “search” and “training” crawler identities or get blocked. Because Cloudflare sits in front of a very large share of websites, its change of default carries more real weight than any self-declared standard.
But this doesn’t change this article’s conclusion: it’s permission control — block vs allow — not a way to get you paid. What to take from it: even the infrastructure layer is now answering “AI scrapes a lot and sends back almost nothing” with a default block. For the full policy detail and what the headlines overstated, see Cloudflare Blocks AI Crawlers by Default.
For small businesses: the point isn’t charging, it’s not blocking the wrong thing
The players actually getting paid by AI are Reddit- and major-media-scale. For the vast majority of sites, what this layer demands now isn’t “how do I charge” but “while setting licensing/blocking, don’t accidentally block yourself out of AI answers” (see Block the wrong bot).
So should you act
Track it, don’t invest. Once the layer consolidates and AI actually starts complying, then talk monetization. For now, energy spent on the site-health that gets you cited beats energy spent on blocking and charging.