Training Data Is the New Ranking Factor: The Common Crawl AI Visibility Audit Explained.

Common Crawl is a nonprofit, founded in 2007 by Gil Elbaz, with one job: keep an open, downloadable copy of the web that anyone can use. It runs a bot called CCBot that crawls the open web every month and publishes the results as free archives on Amazon S3.

Here is why that matters to AI search. Those archives became one of the core training sets for modern large language models. When OpenAI and the other labs started building, Common Crawl was one of the first datasets they reached for. So the chain is short and brutal: block CCBot and you block yourself out of the data these models learn from.

The scale tells you why a single blocked crawler costs so much.

Four steps separate a published page from a model that knows it exists. You can only act on the first one. Most SEOs have never looked at it.

The guide's central claim follows from that chain: training-data inclusion behaves like a ranking factor. It sits upstream of on-page work, technical SEO, and link building, and infrastructure decisions set it (CDN configuration, robots.txt rules, server-side rendering) rather than content strategy.

A model can know about you in two completely different ways, and they have nothing in common except the word "visibility". One is baked into the model's weights. The other is fetched live, on demand. They answer to different crawlers, on different clocks, and a site can win one while losing the other.

The trap is assuming one covers the other. A site can be wide open to CCBot for training and still block the live bots that serve RAG answers. A full picture checks both clocks.

AI crawlers are now the most-blocked agents on the web. Most of those blocks were never a decision. They were a default.

A page can rank well in Google and still be unreachable by training crawlers, because the block lives in the infrastructure, not the content. Two mechanisms cause nearly all of it, and both are invisible from inside your own files.

• Managed robots.txt. Some CDNs inject AI-crawler disallow rules into your robots.txt for you. They create the file if it doesn't exist, or prepend their own rules to the top of one that does, disallowing GPTBot, ClaudeBot, Google-Extended, and CCBot. Your repo looks clean. The block is happening at the edge.
• WAF and bot-management blocks. A "block AI bots" toggle or a bot-fight rule rejects requests by user agent at the firewall, before they reach the server at all. Again, nothing in your files. The bot just sees a 403 and leaves.

This is the default state, not a fringe case. One major CDN sits in front of a huge slice of the web, and for new domains its out-of-the-box setting blocks AI crawlers. Nobody opted in. The setting opted in for them.

The practical takeaway is blunt. For any publisher or brand, assume CCBot access is already gone and check it first.

Open access gets you crawled. How deeply, and how often, comes down to where you sit in the link graph. Common Crawl publishes a Web Graph mapping the web's link structure at host and domain level, and from it derives a measure called Harmonic Centrality.

PageRank and Harmonic Centrality ask different questions. PageRank counts how many important sites link to you: popularity. Harmonic Centrality measures how close you sit to the core of the web's link topology: proximity. A domain near that core gets prioritised in the crawl budget. A domain stranded at the edge gets crawled shallowly and rarely, even with respectable PageRank and a wide-open robots.txt.

One link from a genuinely core site can lift your centrality more than dozens from isolated ones.The AI Visibility Audit, on what link building is really for now

That changes the brief for link building. The old question was "how many authoritative links does this site have?" The new one runs alongside it: "how connected is this site to the core of the graph the training crawler prioritises?" A site can score well on the first and badly on the second. Higher centrality means crawled more often, which means more pages in the archive, more training data, and a better chance the models know you and recommend you. Each crawl compounds the last.

If you operate in a non-English market, here is an uncomfortable one: a page can rank well at home and still lose the AI citation to its English equivalent. Two biases stack to make it happen.

Start with the raw mix. English makes up roughly 41% of the latest crawl (40.85% of CC-MAIN-2026-21, from Common Crawl's own language distribution). After quality filtering the effective share is higher still, because the filter favours well-structured, high-authority pages, and those skew English.

On top of that base, two effects compound. Retrieval bias comes first: AI retrievers rank English and high-authority pages above equivalent localised ones, and it gets worse when the query itself is in or normalised to English. Across multilingual RAG systems the retriever, not the generator, is the bottleneck in cross-lingual ranking. Model bias comes second: large models appear to compute in a shared concept space that tilts toward English, so even parametric answers lean that way.

So pull the lever that actually moves: publish a strong English edition of your cornerstone pages. The retriever is where the bias is most fixable, and a genuinely useful English version of your highest-value pages widens your AI footprint. That means proper hreflang, distinct crawlable URLs, and real content rather than machine-translated filler. Both versions still have to clear the access checks below.

Five checks, in order, most decisive first. Free tools throughout, about 90 minutes start to finish. Run them in sequence: an early failure usually explains the ones that follow.

The five results collapse into a one-page scorecard: pass or fail on access and rendering, present or absent in the archive, a centrality grade, and a structured-data gap list, each with its own fix. Most agencies don't offer that deliverable yet. That gap is the opportunity.

If you read server logs, know this: the CCBot user-agent string proves nothing. Common Crawl says so itself. Other crawlers routinely identify as CCBot when they aren't.

The real CCBot runs from dedicated IP ranges with reverse DNS, so a forward-confirmed reverse DNS check settles it. The IP should resolve to a hostname ending in .crawl.commoncrawl.org, and that hostname should resolve back to the same IP. An impostor fails the round trip. Common Crawl even publishes its verified ranges as JSON at index.commoncrawl.org/ccbot.json. The rule is simple: verify by IP, never by user-agent string, before you touch an access rule or trust a crawl-volume number.

It cuts both ways. Impersonation can inflate apparent CCBot traffic until a site over-blocks and locks out the real training crawler. Or the real CCBot gets banned at the WAF, blamed for load an impostor generated.

The audit is written for sites that want into the training data. It's also honest that plenty of sites don't, and for good reasons: principle, commercial strategy, or outright objection to how the content would be used. The same checks run in reverse for them. They confirm you're genuinely excluded rather than just assuming you are.

The tooling for opting out is maturing. Common Crawl published an Opt-Out Registry in September 2025, a single place to register preferences (including for content already sitting in older crawls), and it's an active member of the IETF AIPREF working group building a richer standard vocabulary for crawler preferences over robots.txt and HTTP headers.

Now the harder truth. Opting out is less reliable than most publishers think. Researchers at ACM CCS 2025 found several thousand domains where the owner believed they had opted out and hadn't. The mechanisms are inconsistent across crawlers: a rule that stops one bot does nothing to another, and intent quietly drifts from outcome. It's the most credible criticism of the current crawling ecosystem, and Common Crawl is working on it directly.

Most GEO frameworks skip a step. They start with content and structure and links, and assume the systems doing the training and retrieving can actually reach the page. The AI Visibility Audit puts that assumption first, where it belongs.

This series has covered how LLMs generate responses (Article 2), how RAG retrieves content (Article 3), how Google's AI Overviews and AI Mode work (Article 5), and what each platform says about getting cited (Article 6). Every one of those depends on access. The Common Crawl audit is the check that confirms it, and for a meaningful share of sites it will come back negative. CDN-injected blocks, WAF rules, JavaScript-only rendering, low centrality: none of these are content problems, and no amount of content strategy fixes them.

If you're not in the crawl, you're not in the model; if you're not in the model, you may not be in the answer. The audit is the free, 90-minute way to find out which side of that line you're on, before you spend another quarter optimising content a crawler never sees.

The old world was index and rank. The new world is train and retrieve.