Blocking AI Training Can Cost You Search: Cloudflare's September Defaults Explained.

Cloudflare announced the change on 1 July 2026. Rather than one switch marked "AI bots", there are now three categories, and you set a policy for each.

The new default is narrower than most of the coverage suggests, so it's worth being precise about who it hits. Cloudflare's wording is that "for all new domains onboarding to Cloudflare, the categories of Training and Agent will be blocked by default on the pages that display ads."

Three qualifiers there matter. It applies to new domains, not retroactively to every existing customer. It applies to pages displaying ads, not your whole site. And it leaves Search alone, which is the category most SEO teams actually care about. If you already run a site on Cloudflare and you haven't changed anything, your settings are your settings.

That's the reassuring part. Here's the part that isn't.

The three categories describe behaviours, not bots. A single crawler can perform more than one of them, and plenty of the important ones do.

Cloudflare is explicit about how it handles that: multi-purpose crawlers "will be allowed/blocked according to all of their behaviors", and the defaults are "enforced by the most restrictive applicable rules". It names the crawlers this affects, and they are not obscure ones. Google, Apple and Bing all run bots that combine search with training.

So the failure mode is this. You decide, reasonably, that you don't want your content training somebody's model. You block Training. The crawler that also indexes you for search is doing both jobs, the most restrictive rule applies, and you have just made yourself harder to find in the surface that still sends you traffic.

You can block training. You just can't always block only training.

This is the same lesson as the CDN-injected robots.txt rules covered in the Common Crawl visibility audit, arriving one layer up. The decision looks like a content policy. It behaves like an infrastructure change, and infrastructure changes have blast radius.

Cloudflare's answer to the mixed-use problem is a compliance test, and it's aimed at the AI companies rather than at site owners. A crawler that does both search and training can avoid the training block, but only by meeting four transparency conditions.

• It must respect a "no training" preference in robots.txt, through any mechanism.
• It must provide a way to opt out of AI summaries.
• It must offer URL-level visibility into which of your pages were made available for training.
• It must demonstrate that disallowing training doesn't damage traditional search visibility.

Cloudflare tracks which bots meet the conditions, publishes the list, and blocks the ones that don't regardless of what your own policy says. That third condition is the interesting one. URL-level reporting of what was used for training is a genuinely high bar, and it's more disclosure than any major platform currently offers publishers voluntarily.

Read that as leverage rather than as a feature. Cloudflare sits in front of enough of the web to make compliance commercially rational, and it is using default settings as the lever. Whether the labs actually meet the bar is the thing worth watching over the next two quarters.

On 21 August 2026 Cloudflare shipped Bot Preference Sync, and it fixes something that has quietly caused a lot of confusion.

Until now, what you enforced at the edge and what you published in robots.txt were two separate pieces of configuration that drifted apart constantly. A firewall rule blocked a crawler that robots.txt welcomed. A robots.txt disallow sat there being politely ignored by a bot the WAF was already letting through. Bot Preference Sync generates and updates robots.txt from your dashboard policy, so, in Cloudflare's words, "what you say to the world and what you enforce at the edge are kept in sync." It prepends its directives and preserves your custom rules, and it's available on every plan including the free tier.

If you audit crawler access for clients, this changes your method. A site's robots.txt is now, for a growing share of the web, a rendering of a dashboard setting rather than a file somebody wrote. When it says something surprising, the question is which toggle produced it.

Four checks, in order. None of them take long, and the first one settles whether the rest matter to you at all.

The honest summary is that this is a narrower change than the headlines suggest and a more interesting one than they convey. Very few existing sites will wake up on the 16th with a problem they didn't have on the 14th. But the category split is now the model everyone else will copy, and the mixed-use rule is a genuine trap that will catch people for years.

If you take one thing from this: stop thinking of AI crawler access as a single decision. It's three decisions, they interact, and the crawlers you most want to keep are the ones most likely to be caught by a rule you set for a different reason.