What AI tools actually read on a web page: a server-log test

Results from the November 2 test, plus a positive control on November 5. A dated update will be added here after a week of daily log checks.

The moment that surprised me most: Perplexity told me it had opened a URL directly, and my server logs show it never sent a request for it.

What each AI tool read

The two prices in the page text came back every time. ChatGPT, Google AI Mode, Claude and Gemini reported Pelican and Osprey in all 12 of their runs, so hiding a price in a closed accordion made no difference here, as long as the text was in the HTML.

Nothing outside the page text came back, including the schema price, which sat in the HTML but only inside a script tag. Heron, Egret, Cormorant and Kittiwake were each reported in 0 of 15 runs. Claude said outright that it had fetched the "server-rendered HTML."

Every tool that reached the page repeated its note that the prices weren't real, and none of them made up a price for the missing plans.

What the server logs showed

The answers only tell you what a tool said. The access log shows what it actually asked the server for. These are the requests for the test page and its files during the test, not counting my own browser.

ChatGPT and Claude each fetched the page once, on the first run. On runs two and three they answered without a new request. The logs can't show where those answers came from. The likeliest explanation is a copy cached from the first fetch, but I can't confirm that. The November 5 control points the same way for Claude: a fresh incognito chat, with no memory, answered correctly with no new request, about 16 minutes after an earlier fetch of the same file.

Google fetched the page live while it wasn't in Google's index. The three Google-Lens requests came from IPs inside Google's published list of user-triggered fetchers, and Google AI Mode's sources panel pointed to the page even though a site: search returned nothing. None of those fetches picked up the JavaScript price.

Nobody but me ever requested the file the JavaScript loaded the Heron price from. No AI agent requested llms.txt or the Markdown copy either. From October 31 until the control on November 5, llms.txt was requested only by SEO tools (Semrush's site audit bot, SEOJuice, Piperic and Keywords Everywhere), a web survey bot, a fetch-cache bot and once by an ordinary Firefox browser on Android.

A user-agent name on its own doesn't prove anything. The morning after the test, a scanner at 35.203.82.17 sent 250 requests in a few minutes, rotating through the ChatGPT-User, Perplexity-User, PerplexityBot and Claude-User names while looking for configuration files and keys. Its IP isn't in Perplexity's published lists. If you count AI bot visits from user agents alone, scanners like this end up in your numbers.

Could they have read llms.txt?

A zero only means something if the tool could have found the price. So on November 5, I asked each tool to open llms.txt and the Markdown copy directly, and checked the access log for each request.

Three of the four tools that read the test page also read llms.txt when I asked. They just didn't check it when I asked about the page. The file was at the standard location and linked to the test page, and no tool went looking.

The Markdown copy was messier. ChatGPT downloaded the whole file, then said it couldn't use it because of its content type. I served it as text/markdown, which is the official file type for Markdown, and ChatGPT's tool turned it down. A site that serves .md files as plain text might get a different answer. Google AI Mode fetched the file three times and reported the right price while saying the URL was inaccessible, so its own account of what it did was wrong.

Gemini failed on both files, and on the HTML test page it had read correctly on November 2, with no request reaching the server. That looked like a problem on Gemini's side that morning, so I count its control as inconclusive, not as a no.

Why Perplexity couldn't see the page

Perplexity only answered from pages its own crawler had already visited. When I gave it a URL it hadn't crawled, it said the page wouldn't load, and it never sent a request to fetch it.

The site wasn't blocking it. Genuine PerplexityBot, from IPs in Perplexity's published range, visited other pages on the site during the test: at 10:07 pm (the services page and robots.txt), around 10:17 pm, and from 10:38 to 10:39 pm. It never asked for the test page, because nothing linked to it.

Perplexity documents a separate agent, Perplexity-User, that visits pages when a user asks about them. That's the one that should have shown up, and there were no requests from its published IPs all day. To rule out my own site, I also set up a request-logging URL on a separate host (webhook.site), which records every request it gets. Then I asked Perplexity five more ways, each in a fresh thread.

None of them produced a request. As a control, I gave ChatGPT the same webhook.site URL, and its request arrived within seconds.

The services page is the telling one. Perplexity's crawler had already visited it, and when I asked Perplexity to open it directly, it said it had opened the URL and answered with the page's content. It did that twice, once with a unique query string added, and no request for that page reached the server either time. It was reading its stored copy.

In these runs, pasting a link didn't make Perplexity read a new page. A page has to be crawled first, which means linked from somewhere PerplexityBot visits. And asking an AI tool whether it read your page isn't a test. Only the logs can tell you that.

Limitations

This is one page on one small site, tested on one evening with a short control three days later, with one account per tool, from Vancouver. ChatGPT was logged out and Perplexity used its default Search mode. The control added AI requests for llms.txt and the Markdown copy to the logs, so the finding that no AI agent asked for them covers October 31 to the morning of November 5. It tests what a tool could read on this page, not how often AI tools cite pages or how common any of this is across the web. The larger studies cover that side, and the results here line up with them: Vercel and MERJ found no JavaScript rendering by the major AI crawlers across their network, Ahrefs found that 97% of the llms.txt files on the 137,000 sites it tracked got no requests in a month, Ahrefs also found no AI citation lift from adding schema, and HasData's AI Crawler Block Index measured how firewalls and robots.txt affect AI crawlers. What this test adds is the request-by-request view from the server.

How to run this test yourself

1. Make an unlinked test page with a visible note that it's a test, and plant one made-up fact in each place you want to check (plain HTML, JavaScript, schema, llms.txt, anything else).
2. Check the page source and the rendered page, and confirm that robots.txt and your firewall allow the AI user agents.
3. Ask each tool the same question in a fresh chat, three times, and save the exact answers and screenshots with timestamps.
4. Ask each tool to open each file directly, so you know a zero means it didn't look rather than couldn't read it.
5. Pull your raw access logs for the test window and match each request to a run by time.
6. Check bot IPs against each company's published range files before you trust a user-agent name.