How AI Search Works
1
Attention Is All You Need: The Google Paper That Accidentally Ended Google's Search Monopoly.
On June 12, 2017, eight researchers at Google Brain published a 15-page paper. It accidentally made Google's own search business obsolete.
2
How Large Language Models Actually Work: Tokens, Context Windows, and Why Your Content Gets Ignored.
The answer to "how do I get cited by AI search?" is technical, not strategic. It's rooted in how large language models actually process information.
3
What Is RAG: the Only Thing That Matters for AI Search.
Once you see how Retrieval-Augmented Generation works, the rest of AI search visibility clicks into place. It's the mechanism that connects your content to every answer these systems produce.
4
The AI Search Race: ChatGPT, Gemini, Claude, Copilot, and Perplexity. A Technical Timeline.
The models, the milestones, and the architectural decisions that shaped every AI search product in market. A reference for anyone trying to make sense of where this came from.
5
How Google AI Overviews and AI Mode Actually Work: Why Your SEO Rankings Don't Guarantee AI Visibility.
AI Overviews aren't a ranking bonus layered on top of existing positions. In May 2024, Google restructured how most people receive information. The blue links became supplementary.
6
How to Be Surfaced in AI Search: What the Platforms' Own Documentation Actually Says.
Most GEO content produces a list of tactics and implies they work across all platforms. This article goes directly to what each platform has actually published about how its own system works.
7
Why the Bing AI Performance Report Is the Most Underused Tool in GEO.
Microsoft's Bing AI Performance report is the only first-party citation dashboard from any AI search platform. Most people working in GEO aren't using it.
8
The GEO Documentation Gap: Why Three Platforms Haven't Told You How to Rank in Their Systems.
Google published a guide. Microsoft published measurement tooling. Anthropic, OpenAI, and Perplexity have published nothing equivalent. Understanding why is more useful than waiting.
9
Constitutional AI vs RLHF: Why the Alignment Method Affects What Gets Cited.
ChatGPT is aligned with RLHF. Claude is aligned with Constitutional AI. These aren't interchangeable, and the difference affects what each model treats as trustworthy and citable.
10
Training Data Is the New Ranking Factor: The Common Crawl AI Visibility Audit Explained.
Before RAG, before indexing, a site has to be reachable by the crawlers that feed AI training data. The Common Crawl AI Visibility Audit is the framework for checking that upstream layer.
11
Blocking AI Training Can Cost You Search: Cloudflare's September Defaults Explained.
Cloudflare split AI bots into Search, Agent and Training. Multi-purpose crawlers resolve to the most restrictive rule, so a training block can take your search visibility with it.
12
Google Finally Reports AI Visibility. It Still Won't Tell You What It's Worth.
Search Console's Generative AI report gives you impressions and nothing else. No clicks, no queries, and no split between AI Overviews and AI Mode. Here's how to use it honestly.
13
The AI Search Statistics Everyone Repeats, and Where They Actually Came From.
Gartner's 25% prediction rests on a survey of 299 people. The 14% AI Mode figure isn't Google's. AI traffic converts 3x, not 23x. The audit, and the method for running it yourself.
14
97% of llms.txt Files Are Never Read. Here's the Data.
Ahrefs studied 137,000 domains and found 97% of valid llms.txt files received zero requests. No major AI provider has adopted it, and Google says you don't need one.
15
Schema Markup Doesn't Drive AI Citations. It Still Matters.
1,885 pages that added schema, against 4,000 matched controls, tested with difference-in-differences. The effect on AI citations was nil. Keep your schema anyway, and here's why.
16
Writing  ·  Series: How AI Search Works  ·  Part 16 of 16

The AI Browser Died. Now Agents Browse as Your Reader.

OpenAI shut down Atlas in August 2026 and moved agentic browsing into a Chrome extension and its desktop app. Anthropic never shipped a browser at all. The winning model puts the agent inside your reader's session, using their logins, and that quietly breaks how most people think about blocking bots.

For about a year the assumption was that AI companies would come for the browser. Own the browser, own the entry point, own the web. It was a clean thesis and the company that pushed it hardest has just abandoned it.

What replaced it matters more than the failure did. Agentic browsing didn't die, it moved: into extensions running inside the browser your reader already uses, with their sessions and their logins. That changes what arrives at your server, and it undermines the idea that you control AI access by blocking bots.

What actually happened.

OpenAI launched Atlas, a dedicated AI browser, in October 2025. It shut down on 9 August 2026, less than a year later.

The capabilities didn't disappear. They moved into a ChatGPT extension for Chrome and an upgraded browser mode inside the ChatGPT desktop app, with multiple tabs, downloads, better navigation and account login support. The conclusion OpenAI reached wasn't that agentic browsing was a dead end. It was that shipping a whole browser to house it was the wrong container.

Anthropic arrived at the same place from the other direction and never built a browser. Claude for Chrome is an extension, in research preview from August 2025 and available across Pro, Team and Enterprise plans by December 2025. It takes actions on your behalf inside Chrome, with site-level permissions and confirmations, working against whatever you're already logged into.

Two companies, opposite starting points, same destination: the agent lives in the incumbent browser, not in a new one.

The consequence nobody put in a press release.

Here's what that architecture means for your server logs.

A traditional AI crawler announces itself. GPTBot, ClaudeBot, CCBot, PerplexityBot: each arrives with a user agent you can identify, from IP ranges you can verify, and you can decide whether to serve it. That whole model assumes the AI is a third party visiting from outside.

An agent running as an extension in your reader's browser is not a third party. It renders your page in their session, with their cookies, their authentication, their consent already granted. From your infrastructure's point of view a substantial share of that traffic is a logged-in human using Chrome, because in every technical sense that's what it is.

You can block a crawler. You can't block your own reader.

This is worth sitting with if you've been advising clients that blocking AI bots gives them control over how AI systems use their content. It gives them control over the crawlers that identify themselves. It gives them very little over an agent operating inside a session they've already authorised.

Google is documenting this openly.

This isn't a fringe reading. Google added a section on agentic experiences to its AI optimisation guide in the 10 July 2026 revision, and it describes the mechanism plainly.

Browser agents, per Google, may access websites to gather data through visual rendering analysis and DOM inspection. In other words they look at the page the way a browser presents it and they read the underlying document structure. The guide also points site owners towards emerging protocols such as the Universal Commerce Protocol for businesses where agents might transact.

That's Google telling publishers, in its own documentation, that an increasingly important consumer of their pages is a piece of software driving a real browser. Cloudflare has reached the same conclusion from the infrastructure side, which is why its new bot taxonomy carries a separate Agent category for real-time automated behaviour on behalf of a user, distinct from Search and Training.

Three things this quietly breaks.

Bot blocking as an AI content strategy. If your plan for controlling AI access is a user-agent blocklist, it addresses the crawlers and misses the agents entirely. That's not an argument against crawler controls, which still matter for training and indexing. It's an argument against believing they're comprehensive.

Analytics. An agent operating in a logged-in session produces sessions that look human, because they are running in a human's browser. Depending on how the agent navigates, that traffic may land in your analytics indistinguishable from a person reading, or may not register at all. Either way, "AI referral traffic" as a measurable category gets less reliable, not more, which is worth remembering the next time someone quotes you a precise figure for it.

JavaScript rendering assumptions. Much of the advice in this series, including in the Common Crawl audit, notes that many AI crawlers fetch HTML without executing JavaScript. Browser agents are the exception: they're driving a real browser, so they see the rendered page. That doesn't make server-side rendering less important, because the crawlers that feed training and indexing still don't render. It does mean the two consumers now have genuinely different capabilities, and advice that treats "AI" as one audience is getting less accurate over time.

What to actually do.

Less than you'd fear, and the work is mostly the same work.

The pattern across this whole series is that the mechanism outlives the product. Atlas is gone and Comet may follow, but the capability moved rather than disappeared, and that capability now sits one layer closer to your reader than any crawler ever did.

The clean version of the old mental model was: search engines send you people, crawlers take your content, and you can tell them apart. That distinction is dissolving, and the honest answer is that the tooling for the middle case doesn't really exist yet.

Frequently asked

What happened to the ChatGPT Atlas browser?

OpenAI launched it in October 2025 and shut it down on 9 August 2026. The agentic browsing capabilities moved into a ChatGPT extension for Chrome and an upgraded browser mode in the ChatGPT desktop app, with account login support.

Can I block AI browser agents?

Not reliably. An agent running as an extension in a reader's browser uses their session, their cookies and their existing logins, so it presents as a logged-in human rather than an identifiable crawler. User-agent blocking addresses crawlers, not agents.

Do browser agents execute JavaScript?

Yes. They drive a real browser, so they see the rendered page, and Google's documentation describes them using visual rendering analysis and DOM inspection. That's different from training and search crawlers, many of which fetch HTML without executing JavaScript.

Does this mean server-side rendering no longer matters?

No. The crawlers that feed training data and search indexes still frequently don't render JavaScript, and they remain the route to being in the model and in the index. Browser agents are an additional consumer with different capabilities, not a replacement for the others.

How do agents show up in my analytics?

Unreliably. Because they operate inside a real browser session, agent activity can look like ordinary human traffic or may not register cleanly at all. That's one more reason to treat precise "AI referral traffic" figures with caution.

Should I be doing anything differently right now?

Mostly not. Keep clean semantic HTML so a machine reading the DOM can parse your page, separate training, search and agent traffic when you plan crawler policy, and stop describing bot blocking as full control over AI access. The share of traffic from agents is still small.

Back to the start

That's the series. If you're new here, the foundations start with the 2017 paper that made all of this possible, and everything else builds on it. Read Article 1 →

SOURCES ↓
  1. OpenAI Help Centre, "Evolving Atlas into ChatGPT for browser-based agentic work" - help.openai.com/en/articles/20001371
  2. Anthropic, "Claude for Chrome" (research preview August 2025; Pro, Team and Enterprise from December 2025) - claude.com/blog/claude-for-chrome
  3. Google Search Central, "Optimizing for Generative AI Features on Google Search," agentic experiences section, revised 10 July 2026 - developers.google.com/search/docs/fundamentals/ai-optimization-guide
  4. Cloudflare, "Your site, your rules: new AI traffic options for all customers," 1 July 2026 - blog.cloudflare.com/content-independence-day-ai-options

Let's talk about your visibility.

Invisible in AI answers, slipping in search, or both. Thirty minutes is enough to work out where to start.

Book a call →