Your Rank Tracker Can't See 62% of What Gets Cited

Google Top 5 gets you less than you think

Here is a number that should ruin an SEO’s week. Profound ran 100,000 prompts through both ChatGPT and Perplexity and found the two share about 11% of the domains they cite. The other 89% depends entirely on which model you happened to ask. Across the platform pairs Profound tested, overlap runs 6% to 16.4%, so 11% is the middle of the road, not a cherry-pick. Ask two assistants the same question, you get two almost completely different reading lists.

And if you spent twenty years getting good at Google, it gets worse. Ahrefs checked 863,000 keywords and 4 million AI Overview URLs. Picture a hundred links Google quotes in its own AI answers. Thirty-eight were sitting in the top 10, the slots you fought for. Thirty-one were buried somewhere back on pages two through ten. The last thirty-one were not ranking anywhere at all.

So roughly two out of every three pages Google quotes never won the fight you were having.

Google messed up. Royally. And in the process they messed themselves up.

While nobody was looking they also did the impossible: they brought back the fold.

I searched best running shoes for flat feet on my own phone and recorded it. From the top of the results page to the first unpaid, non-Google-written result sitting at the top of my screen: 1,805 points of scrolling on a 932-point screen. Just under two full screens. Above it, in order: a shopping carousel, a block of sponsored results, a “hide sponsored results” toggle, and Google’s AI Overview answering the question itself.

The detail that stopped me was further down the same page. Runner’s World appears twice. Once inside the sponsored block, and again as the first organic result. The publisher that earned the slot is also buying it.

I spent eleven years at Hearst and Runner’s World is one of the titles I worked on. Good years, good people, and buying that slot is the right call. Which is exactly why the screenshot stings. Nobody there did anything wrong. The page changed underneath them, and the only lever still fully under a publisher’s control is the one with a price on it.

The blue links did not lose to the AI agents. They lost to the answer box sitting on top of them, which Google built.

Small confession: the 62% moved while I was writing this. An earlier draft said 53%. I could not find a real source for 53%, so I replaced it with a number I could source. Other trackers land on different splits, but they all point the same way, and the direction is the whole point: Google’s answer engine disagrees with Google’s search engine most of the time.

So the top slot, the thing an entire industry exists to capture, is now one input to one of five systems. Ranking is not dead: run the per-slot math on Ahrefs’ own numbers and a top-10 slot still gets cited about eleven times more often than one at 11 to 100. It is just no longer the game. Most of what gets cited comes from somewhere your rank tracker never looks.

The king’s own two engines stopped taking orders

For twenty years, “being findable” meant one company’s index and one ranking algorithm. That is done. There are five answer engines I could verify retrieval paths for, running on four different setups, and only two of them look at your Google rank at all.

Even the two engines running on Google’s own index cannot agree with each other. Ahrefs put 730,000 of the same questions to both: near-identical answers, 86% semantic similarity, and only 13.7% overlap in the URLs they cite. Same company, same index, two different reading lists.

The game reset and most of the industry has not noticed. The old dashboard still shows numbers, and the numbers still look fine in a meeting. This is fine.

Statistics, quotes, and citations beat keywords

One content change has a positive result from a controlled experiment: evidence density. The GEO paper from KDD 2024, GEO being generative engine optimization, built a 10,000-query benchmark, edited the source pages, and scored how much of a GPT-3.5-generated answer each source won, with a smaller check on Perplexity. Statistics, quotes from credible sources, and citations bought up to about 40% more visibility. “Up to” is doing some work in that sentence, but the direction held.

Keyword stuffing, the old religion, did nothing at all: +0.3% overall, and negative only for sources that already ranked first. Pour one out.

One caveat: a critical survey of the GEO literature points out the 40% is conditional on your page already being retrieved. It moves your share of the answer, not whether you are in the answer at all. The same survey re-scores the paper’s gains lower too, 22% to 37% depending on the metric. That is a narrower claim than the vendor blogs make, and a more believable one, twice over.

The agents reward the exact thing the old content economy spent twenty years punishing: showing your work.

For twenty years SEO was a picnic. Show up, spread a blanket, everybody eats. Then Google’s AI decided the picnic was an all-you-can-eat buffet and started serving everyone’s work back from its own table. Now there is no gatekeeper. There are five tables, and the only way in is bringing something worth serving.

Receipts beat keywords. Bring sources or stay home.

And yes, this post is built that way on purpose, which is either good practice or a snake eating its own tail, depending on how generous you are feeling. Every number in it shows up in the receipts at the bottom with its grade and its source, or with a tag saying I went looking and could not find one. If the thesis is right, this page should do well. If it does not, that is a data point too, and it goes in the table.

What the AI agents actually read

No JavaScript

Vercel watched 1.3 billion AI crawler fetches and every major AI crawler except Google’s and Apple’s executed zero JavaScript. OpenAI’s, Anthropic’s, Meta’s, ByteDance’s and Perplexity’s crawlers all download your JavaScript and none of them run it. On Vercel’s own docs site, JavaScript files were 11.5% of ChatGPT’s fetches and 23.8% of Claude’s, fetched in full and executed not at all. Everything your framework does after hydration is invisible to them. All your client-side rendering are belong to nobody. Whatever is not in that first response might as well not be on the page.

Other people’s pages, not yours

Mentions correlate with AI visibility about three times more strongly than backlinks do: Spearman 0.664 against 0.218, Ahrefs, 75,000 brands. The roughly-3x framing is Ahrefs’ own, and their sample is domains above DR 40, so it describes established brands more than it describes a site like yours. You have also heard that 82 to 94% of AI citations go to earned media instead of brand sites. Everyone repeats it. I went looking for the actual study and came up empty Unverified . That stat is doing keg stands at every marketing conference and nobody knows who invited it.

Tables and lists

Pages with tables supposedly get cited about 2.5 times more than straight prose Unverified . Quoted constantly, linked to a real study almost never. Ironic, given the topic.

Fresh pages

Perplexity supposedly will not touch anything older than six months Unverified . Recency bias in AI citation is real and widely reported. That specific threshold is folklore.

Alt text

On Vercel’s own docs site, images were 35.2% of ClaudeBot’s fetches, same Vercel dataset, Dec 2024. That number says the crawlers want your images. It does not say they read your alt text, and I went looking for a study that measures whether they do and came up empty Unverified . Write it anyway. Someone using a screen reader is a better reason than a bot, and it happens to be the only part of an image either of them can read.

I once had to do something stupid to support Internet Explorer, because at our scale 1% of 300 million is a statistically significant number. The columns in a grid were slightly out of alignment, because we numbered them starting at zero, which makes sense to an IE6 developer or a CSS-Grid spec with no design experience. I wrote two lines of SCSS, a negative offset of about 1.25rem, to put the columns back where the design had them. It felt gross. It doesn’t sound serious, but re-checking 30 sites on three different devices by hand is not fun. The lead developer told me to let it go. I didn’t know he was building the same components in React with CSS-in-JS, which wasn’t my chair and wasn’t my problem, and from inside that stack two lines of SCSS meant nothing. Nobody was being careless. Small semantic decisions get overruled that way constantly, by people whose reasons hold up fine in the moment.

I tell you that because the reader changed and the physics did not. The thing reading your page now does not skim past the small semantic stuff. The small semantic stuff is the only stuff it gets.

The GEO graveyard

llms.txt is dead. Ahrefs analyzed 137,000 domains: of the 38,360 that actually published an llms.txt, 97% never had it fetched in a month of logs. I could not find a controlled study of its citation effect in either direction, which is unsurprising when almost nothing reads the file. And Google’s own guidance says flat out that you do not need llms.txt or any special AI file to appear in Google Search, including its generative features, “as Google Search itself doesn’t use them.”

Between them, llms.txt and robots.txt are the most published, least read files on the internet. Yours is not winning a Pulitzer either. Good idea, nobody ever built the reading side. It died as it lived: unread.

Schema markup is not a growth hack. Ahrefs tracked 1,885 pages that added schema against roughly 5,700 matched control pages. Citations barely twitched: +2.4% in Google AI Mode, +2.2% in ChatGPT, and negative 4.6% in AI Overviews. Ahrefs’ own read is that the two positives are statistically indistinguishable from zero. The only number that reached significance was the negative one. Keep your schema anyway, because it is cheap and it does other jobs well. Just do not expect it to move this needle.

The robots.txt fine print nobody reads

The vendors split their bots into fleets, and almost every robots.txt in the wild gets this wrong.

GPTBot and ClaudeBot are the training crawlers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot are the ones that get you surfaced. Block training and you are still in AI search. Block search while allowing training and you have cut yourself the worst deal available: they learn from your work and never send a soul your way. This is straight from the vendors’ own docs: OpenAI, Anthropic, Perplexity.

Two details that get flattened everywhere else. Perplexity documents no training crawler at all, and says plainly that PerplexityBot “is not used to crawl content for AI foundation models,” so the tidy training-versus-search pairing does not actually apply to them. And OpenAI runs four bots on that page rather than two, including an ads crawler, OAI-AdsBot.

Then there is a third category, and this one is fun. All three vendors run a user-triggered fetcher, the thing that goes and gets a page because a human just asked a question, and all three say something different about whether your robots.txt applies to it.

Perplexity is the blunt one. Of Perplexity-User: “Since a user requested the fetch, this fetcher generally ignores robots.txt rules.” Their word, ignores. OpenAI hedges, saying ChatGPT-User “is not used for crawling the web in an automatic fashion” and that “because these actions are initiated by a user, robots.txt rules may not apply.” Anthropic claims the opposite outright, that “Anthropic’s Bots respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt,” with no carve-out for Claude-User, which it now documents alongside ClaudeBot and Claude-SearchBot.

So on one question, one vendor ignores your file, one says it might not count, and one says it always does. A human asks an assistant a question, and the assistant goes and grabs your page. Whether your crawl directives apply to that fetch depends entirely on whose assistant it was. At that point robots.txt is a strongly worded letter.

Which makes those hits the best free measurement you have. Every verified user-fetcher hit in your logs, verified meaning the vendors’ published IP ranges rather than a spoofable user-agent string, is a request made on behalf of somebody’s actual question. All three publish JSON IP lists for exactly this, OpenAI, Anthropic and Perplexity, and Perplexity is the only one that documents how to wire them into a WAF rule. None of the three documents reverse DNS verification, which is the Google-style check most people assume is available here and is not. Not an impression. Not a rank. Somebody asked something, and an assistant went and got your page. Whether it ended up cited, or read, is a separate measurement.

I am not speculating about that last part. Perplexity is how half the sources in this post reached me in the first place. I asked it questions, it went and fetched somebody’s page, and that is the only reason I am linking them here.

The receipts

Every claim on this page, its grade, and where it came from.

Measured

ChatGPT and Perplexity share about 11% of the domains they cite, and across the platform pairs tested the overlap runs 6% to 16.4%. Profound, 100,000 prompts, Jun 2025.

Measured

Of the URLs Google’s AI Overviews cite, 38% also appear in the top 10, 31% rank 11 to 100, and 31% do not rank at all. Ahrefs, 863,000 keywords and 4 million AI Overview URLs, Mar 2026. The 38% counts all SERP blocks; organic-only it is 37.1%.

Inferred

A top-10 slot gets cited about eleven times more often than a slot at 11 to 100. My arithmetic on Ahrefs’ own splits above, not a figure they published.

Measured n=1

1,805 points of scrolling from the top of the results page to the first organic result, on a 932-point screen. My own iPhone, Safari, best running shoes for flat feet, 2026-09-10, screen recording measured frame by frame at 20fps by optical offset matching. One query, one device, one moment, not a survey.

Measured

AI Mode and AI Overviews cite the same URLs only 13.7% of the time, at 86% semantic similarity. Ahrefs, 730,000 response pairs, Dec 2025.

Measured

Evidence density lifts a source’s share of generative answers: quotations about 41%, citing sources 30 to 40%, statistics 28%, on an LLM-judged metric. GEO paper, KDD 2024, 10,000-query GPT-3.5 benchmark.

Analytic

The 40% is conditional on the page already being retrieved, and the survey re-scores the paper’s gains at 22 to 37%. Critical survey of the GEO literature, unrefereed preprint, Jul 2026.

Measured

Keyword stuffing does not improve generative visibility: +0.3% overall, with harm only among sources that already ranked first. GEO paper, same benchmark.

Measured

AI crawlers other than Google’s and Apple’s execute zero JavaScript. Vercel, about 1.3 billion AI crawler fetches, published 17 Dec 2024, silently modified 6 Feb 2025, the most recent public measurement I found. Named non-renderers: OpenAI, Anthropic, Meta, ByteDance, Perplexity, Common Crawl. One caveat Vercel does not state: the sample is its own network and a developer docs site, so the content mix is not the web’s. Its own conclusion says “nearly 1 billion” where its body says 1.3 billion.

Measured single site

JavaScript files are 11.5% of ChatGPT’s fetches and 23.8% of Claude’s, and none of it is executed. Vercel, Dec 2024, nextjs.org only, those two crawlers only. No JS-fetch share published for Meta, ByteDance or Perplexity.

Unverified

82 to 94% of AI citations supposedly go to earned media, not brand sites. Searched 2026-08-25 and 2026-08-28, no primary study found.

Correlational

Mentions correlate with AI visibility about three times more strongly than backlinks do (Spearman 0.664 against 0.218). Ahrefs, 75,000 brands, DR above 40, May 2025.

Unverified

Pages with tables supposedly get cited about 2.5 times more than prose. Searched 2026-08-25 and 2026-08-28, no primary study found.

Unverified

Perplexity supposedly rarely cites anything older than about six months. Recency bias in AI citation is real and widely reported, but this threshold has no primary source after two hunts.

Measured single site

ClaudeBot spends 35.2% of its fetches on images. Vercel, Dec 2024, nextjs.org only, not the three-property set. ChatGPT’s crawler was 57.7% HTML on the same site.

Unverified

AI crawlers read your alt text. Image fetching is measured; whether the text alternative is parsed is not. Searched 2026-09-16, no primary study found.

Measured

Of the 38,360 domains that published an llms.txt, 97% got zero fetches of it in the study month. Ahrefs llms.txt study, Jun 2026.

Not measured

No public citation-impact study of llms.txt exists in either direction. The Ahrefs study counts fetches, not citations, and two searches (2026-08-28 and 2026-09-16) turned up no controlled citation test.

Vendor documented

Google Search does not use llms.txt-style files: they “aren’t needed” and Search “doesn’t use them”. Google AI optimization guide, updated Jul 2026.

Measured

Schema barely moves citations: +2.4% in AI Mode and +2.2% in ChatGPT, both indistinguishable from zero, and -4.6% in AI Overviews, the only significant number. Ahrefs, 1,885 adopters against about 5,655 matched controls, observational difference-in-differences, May 2026.

Folklore

Vendor blogs quote precise ranking weightings. No methodology published.

Reported not vendor confirmed

Claude’s web search appears to be powered by Brave Search. TechCrunch, Mar 2025, never confirmed by the vendor. Anthropic separately documents Claude-SearchBot as its search-indexing bot.

Vendor documented

Training crawlers and search crawlers are separate fleets. OpenAI, Anthropic, Perplexity, all three read in full 2026-09-16. Perplexity documents no training crawler at all. OpenAI documents four bots, not two, including OAI-AdsBot.

Vendor documented quoted verbatim

The three vendors disagree on whether robots.txt binds a user-triggered fetch. Perplexity: “generally ignores robots.txt rules”. OpenAI: “robots.txt rules may not apply”. Anthropic: “Anthropic’s Bots respect ‘do not crawl’ signals”. Read 2026-09-16.

Vendor documented

Anthropic documents Claude-User and claims robots.txt compliance across its whole fleet. Anthropic, page dated 7 Apr 2026. The archived pre-rewrite copy, captured Oct 2024, documented only ClaudeBot and honored CCBot disallows. Both of those changed, and the doc moved off support.anthropic.com.

Vendor documented by absence

No vendor documents reverse DNS bot verification. JSON IP-range endpoints, all live 2026-09-16: OpenAI search, OpenAI user, OpenAI GPTBot, Anthropic, PerplexityBot, Perplexity-User. None of the three docs mentions reverse DNS, PTR records, or forward-confirmed rDNS.

Grade your own draft

Copy the prompt below and paste it into your assistant, then send your draft’s URL as the next message. You get back every claim you made, what grade it earns, and the three fixes worth making, before your readers find the weak ones for you.

If you make things

The whole shift fits in one line: the agents reward the thing the old economy punished.

Show your work. Cite the source in the sentence that makes the claim. Ship HTML that arrives in the first response. Get talked about on sites that are not yours. Write alt text like someone is reading it, because something is.

None of this is a growth hack. It is making pages that are honest about where the information came from, which is exactly what the last twenty years trained an entire industry out of doing.

So keep it real. Do more uploading and less downloading. Sharing is caring, and it always was. The internet can be cool and weird and worth reading without rewarding genuinely sick behaviour, and the only thing that has ever made that true is people putting real work where other people can find it.