LLM visibility tracking: why your AI visibility score can mislead you

I recently removed four prompts from an AI visibility report I manage, and their score went from 24.76% to 49.64% overnight. Nothing about what ChatGPT said had changed. The brand showed up in the same answers, in the same spots, and I'd only changed the math underneath.

If you're paying for AI visibility tracking, or an agency sends you an AI visibility score every month, that number can go up or down without a single extra customer seeing your brand in ChatGPT.

Before you celebrate a jump or worry about a drop, check three things: which questions are being tracked, whether they changed since the last report, and which AI tools are included.

I've been running LLM visibility tracking in Semrush for a local agency client since July 2026. The examples below come from that account (including a change of my own that could have been misleading), but the same problems can come up in any tracker.

What does LLM visibility tracking measure?

It measures how often AI tools mention or cite your brand when they answer a set of questions. LLM stands for large language model, the technology behind tools like ChatGPT and Gemini. A tracker sends a list of questions (called prompts) to tools like ChatGPT, Gemini and Google AI Mode on a schedule, usually once a day, then records whether your brand is named in each answer and whether any of your pages are used as a source.

It's closer to a poll than a Google ranking. A ranking is one position for one keyword. An AI visibility score is more like asking a group of people a set of questions and counting how many of them bring you up, so if you change the questions or ask fewer people, you'll get a different result even when nobody's opinion has changed.

Two terms matter for the rest of this post:

• Mention: the AI answer names your brand, for example “A few well-reviewed agencies include Acme Marketing and…”
• Citation: the AI tool uses one of your pages as a source for the answer, usually shown as a link.

You can get one without the other. An AI tool can recommend you by name based on someone else's article, or use your page as a source for a fact without recommending you at all. If you're newer to how AI search works, I cover the basics in SEO vs GEO.

The prompts are tests, not real customer questions

The prompts in a tracker aren't what your customers typed into ChatGPT. They're test questions the tool sends itself to see what ChatGPT would answer, usually from a fresh session with no chat history. A real customer might phrase the question differently, and their answer can change based on their past chats and location, so a tracker shows you a sample of what AI tools say, not what every buyer sees.

No AI tool shares your customers' actual prompts with you, but you can get closer. Bing Webmaster Tools' AI Performance report shows how often Copilot cites your pages and the short topic phrases it cited them for, your analytics tools can help show how much traffic you get from AI tools, and asking new leads how they found you fills in a lot of the rest.

You can also work backwards from your own data, by filtering Search Console for long, conversational queries or checking your server logs for the pages ChatGPT fetches when someone asks it a question.

Why can the same brand get different AI visibility numbers?

Because two reports can be asking different questions, counting different things or checking different AI tools.

Different prompt sets give different numbers

Semrush's AI Visibility toolkit checks your brand against its own database of more than 317 million AI prompts. Semrush picks which of those are relevant to you, and you can't choose them. Its Prompt Tracking tool only checks the prompts you add yourself, which for this client was 8 at the time. In October, the toolkit counted 12 mentions and Prompt Tracking counted 2, because the toolkit was looking at far more questions. (The toolkit also has its own 0 to 100 “AI Visibility Score” that compares you with competitors, which is a different number again.)

So why use Prompt Tracking at all? Because you choose the questions. The toolkit is useful for a broad trend and for comparing yourself with competitors, but its prompts can include questions your buyers would never ask, and you don't control whether that set changes from one month to the next. Prompt Tracking covers fewer questions, but it tells you whether you show up for the ones your buyers actually ask, and it gives you a clean before and after.

Mentions and citations can tell different stories

Being named and being cited don't always go together. One of the prompts I track counted as a mention even though ChatGPT didn't link to any of the brand's pages. Over the summer, the brand's mentions in Semrush's toolkit went up while its citations went down, so depending on which line you put in a report, it looked like either a great quarter or a worrying one.

Every AI tool gives different results

The brand's visibility score was 7.42% on ChatGPT. On Google AI Mode it was 0%, with no mentions at all, and Gemini never cited it. A single “AI visibility” number that blends them together, or only shows one of them, hides most of that.

How is an AI visibility score calculated?

Semrush shows two scores in Prompt Tracking, “AI Visibility” and “Visibility,” (not confusing at all 🤦‍♀️) and doesn’t publish how either is calculated, so I compared them with the per-prompt data in its API. Both treat each prompt you track as an equal share of 100%.

The citation-based score, Visibility, depends on where an AI platform lists your page among its sources: first place earns the full share, second earns about a third and 13th about 4%, while not being cited earns nothing.

The default score, AI Visibility, ignores position. Any prompt where the AI platform names your brand earns nearly its full share, cited or not. Add up the shares and you get each score.

Think of it like a quiz where every question is worth the same number of points. With 8 questions, each one is worth 12.5 points, and with 4, each one is worth 25. That's what happened with my report. I was tracking 8 prompts on ChatGPT, and the brand was doing well on 2 of them, for a score of 24.76%. Four of the prompts had never mentioned the brand on any AI tool (too broad for the client to target), so I removed them. With 4 prompts left, the score jumped to 49.64% for exactly the same answers.

In Semrush, two things break when the prompt set changes:

• Before-and-after comparisons: if the prompts change between two dates, the scores aren't measuring the same thing anymore, so part of the change (or all of it) comes from the prompts.
• The history itself: when you remove prompts, Semrush warns that “You will lose all rankings history for these prompts” and recalculates the overall visibility history without them, so last month's score may not exist anymore in the form you reported it.

For comparison, here's what a real improvement looks like. One prompt that had never cited the brand started citing its homepage as the top source. The prompt set didn't change but the answer did, so that change is a real improvement.

What else makes AI visibility data unreliable?

A few smaller things can make a trend look different from what's really happening in AI answers:

• Brand matching: trackers decide what counts as your brand by matching name variants. Semrush automatically counted two unrelated organizations with similar names as the brand I track, which would have inflated its mentions if I hadn't checked. You can review and edit these in Semrush's Manage brands settings.
• Run-to-run variation: each prompt runs once a day on each AI tool, and AI answers change from one run to the next, so a single day is a snapshot. With 8 prompts, one answer changing moves the score by 12.5 points.

A mention isn’t always good news

A visibility score counts a mention the same way whether the answer recommends the brand, lists it in passing, or warns people away from it. “Avoid X if you need fast support” adds to the score just like “X is the best choice for small teams.”

Many trackers add a sentiment score to cover this, but it’s another automated label on top of the same prompt set. It has every problem above, and the classifier can read a lukewarm mention as positive. Before trusting the label, read a sample of the answers that mention you.

How can an AI visibility report make results look better than they are?

In lots of small ways, either by accident or on purpose to inflate the numbers. We all love a little flattery, but I like to keep that sh*t out of my reporting. Finding out where there's real room for improvement is a lot more useful than the dopamine hit that comes from vanity metrics. Each of these can push a score up without any more customers seeing you in AI answers:

• Niche prompts that only you could win, like a very specific service in a very specific town.
• Branded prompts that include your company's name, so of course the answer mentions you.
• Dropping prompts that never mention you, which raises the score without anything improving.
• Changing the prompt set partway through the reporting period.
• Reporting the one AI tool that looks good, like ChatGPT at 25%, and leaving out Google AI Mode at 0%.
• Switching between mentions and citations depending on which went up that month.
• Percentage changes from a tiny base, where going from 1 mention to 3 becomes a 200% increase.
• Best-day snapshots instead of averages over the month.
• Loose brand matching that counts other organizations as you.

I've done one of these myself (by accident, obviously). Removing those four prompts was the right call, because they were broad terms like “shopify web design” that a local agency wasn't likely to win, and the 10 I added in their place are questions a local buyer would actually ask. But if I'd put the new score next to the old one, it would have looked like a big jump in visibility that had nothing to do with AI answers. So I reset the baseline that month, and the number I now report each month is a simpler one: how many of a fixed set of prompts mention the brand.

How should you set up LLM visibility tracking you can trust?

Pick your prompts before you look at any results, keep them fixed for the reporting period and report the raw counts next to any percentage. Whether you only want to track brand mentions in ChatGPT or across every AI tool, I'd set it up the same way:

• Choose prompts from real buyer questions, like the ones that come up on sales calls, in customer emails or in your own team's inbox. Do this before you see any results, so you're not picking prompts you already know you win.
• Keep the set fixed for the reporting period. If you need to change it, note the date and start a new baseline.
• Review your brand name variants before you trust any mention counts.
• Track mentions and citations separately, for each AI tool.
• Report a mention rate with a fixed denominator: the number of prompts that mention you, divided by the number of prompts you track. “5 of 20 prompts (25%)” is much harder to misread than a weighted score.
• Look at monthly trends rather than single days.
• Check which third-party pages AI tools cite for your prompts. In Semrush, the Sources tab shows those pages and whether they mention you, and for the account I manage, one of the most useful finds was a local “best SEO companies” roundup that already mentions the brand. That points to off-site work rather than more changes to your own site, which I cover in Building off-site authority and Are listicles worth it?

What should you ask before trusting an AI visibility report?

Ask how the number was made. These are the questions I'd ask an agency, a tool or myself before reading anything into a score:

I'd also ask to see the list of prompts. A report that can't show you the questions behind the score isn't telling you much.

And an AI visibility score is only one signal. To see whether showing up in AI answers is turning into business, I'd look at it next to the bigger picture: branded searches in Google Search Console, traffic from AI tools in Google Analytics, overall site traffic, and leads and pipeline (including how new leads say they found you). If the score is going up and none of those are moving, I'd want to know why before calling it a win.

Not sure what your AI visibility numbers mean?

I can review your AI visibility setup or a report you've been sent, check how the numbers are built and set up tracking you can compare from one month to the next. See how I work with in-house teams and agencies, or get in touch to talk through your numbers!

‍