What happened

On 15 December 2025, Judge Sidney H. Stein of the U.S. District Court for the Southern District of New York dismissed one of Ziff Davis’s claims against OpenAI: that OpenAI’s crawler ignoring its robots.txt amounted to circumventing a technological protection measure under section 1201 of the DMCA.

The judge didn’t buy it, on the grounds that robots.txt does not “effectively control access” in any technological sense. His analogy was blunt: a file asking crawlers not to take certain content controls who walks onto a lawn no better than a sign reading “keep off the grass.”

What needs stating precisely is that only that one claim fell. In the same order, contributory infringement and the DMCA section 1202 claim over copyright management information were allowed to proceed; unjust enrichment and part of a trademark dilution claim were dismissed. Copyright infringement itself, breach of contract, terms of service — those routes are all still open. The court did not say crawlers may take whatever they like.

Why it matters to you

We’ve written a whole set of robots.txt guides, and throughout them we’ve called it a gentleman’s agreement. Now there’s a judicial characterisation to go with that: its force comes from the other party choosing to comply, not from any ability to stop them.

That cuts two ways, and for you they point in opposite directions.

One side is bad news. If you assumed that writing Disallow into robots.txt amounted to locking a door — something you could sue over — this order tells you plainly that it doesn’t. What you can actually argue runs through copyright and contract, and that fight is about how the content was used, not about a file being ignored.

The other side matters more. Since the power of robots.txt comes from voluntary compliance, and the major AI companies do in fact comply, it stops being a technical safeguard and becomes a switch you operate deliberately. It can’t stop anyone who doesn’t want to honour it, but it stops absolutely everyone who does — including the search and citation crawlers you actually want to let in.

That’s the trap most brands fall into. Not having content taken by a rogue crawler, but blocking OAI-SearchBot, Claude-User and PerplexityBot — the ones that handle live citation — and disappearing from AI answers altogether. The file carries no legal force. It carries absolute force over your own visibility.

What to do about it

One thing not to do: don’t remove your robots.txt because of this order. It remains the one universal entry point for managing crawler access. Just stop treating it as a legal lock.

What to do is work out who you’re actually blocking. Training crawlers and search-or-citation crawlers are different animals, and the cost of blocking the wrong one isn’t content being taken — it’s not existing in AI answers. The full breakdown and a configuration you can copy are in AI crawler rules.

Worth noting too: the legal line is still moving. The same judge also has the New York Times case against OpenAI and Microsoft on his docket. Nobody can tell you where it lands, and anyone promising you an outcome is selling something — but however a court rules, it won’t go back and decide for you which bots to let through. That part has always been in your hands, and most sites currently have it wrong.

If you’re unsure whether your configuration lets them in or shuts them out, we can take a look first.

Further reading