Key takeaways
- AI Overviews and other AI answers are not a stable, ordered list like classic Google results. The same prompt can return different brands, different order, or no AI Overview at all, run to run.
- SparkToro's January 2026 research found less than a 1-in-100 chance that ChatGPT or Google's AI gives the identical list of brands twice, and roughly a 1-in-1,000 chance of the same order twice.
- Tools disagree partly because of randomness in the model, and partly because of methodology: different prompt wording, different sampling frequency, different engines queried, different definitions of a "citation."
- Promptwatch's own data shows average citations per response vary by engine (roughly 5 for ChatGPT vs. about 10 for AI Overviews and Perplexity) and can drop 27% overnight after a model update, which alone explains a lot of tool-to-tool variance.
- Don't trust a single snapshot from any tool. Look for trends across weeks, check raw answers (not just a blended score), and triangulate with manual checks in incognito mode.
Two dashboards, one keyword, two completely different stories
I've seen it happen more than once: a client pulls up two AI visibility dashboards side by side, types in the same brand and the same query, and gets two different pictures. One tool says they're cited in the AI Overview for "best project management software" 40% of the time. The other says 12%. Nobody did anything wrong. Nobody's lying. This is just what happens when you try to measure something that was never built to be measured consistently.
Traditional rank tracking had this problem too, just a smaller version of it. Location, device, and time of day could shift a Google ranking by a few spots. Advanced Web Ranking has written about how this got worse after Google's May 2026 core update, pointing to hyper-localization, a dual-SERP reality where bot traffic and human traffic can see different pages, and NavBoost-style re-ranking based on aggregate clicks. Even a perfect automated tracker can get a page that a real human never would have seen.
AI Overviews take all of that instability and multiply it. There's no fixed list to disagree about the order of. There might not even be an AI Overview at all on the next run.
The randomness problem is bigger than most people assume
SparkToro partnered with a researcher at Gumshoe.ai to actually test this, because nobody had. Their conclusion, stated bluntly: "any tool that gives a 'ranking position in AI' is full of baloney." The numbers behind that statement are worth sitting with.
Ask ChatGPT or Google's AI for a list of brand recommendations 100 times, and you'll get the same exact list less than one time in a hundred. Get that list in the same order twice? About one time in a thousand, according to Searchable's summary of a similar study with nearly 3,000 runs. There's one counterexample worth noting: City of Hope hospital showed up in 69 of 71 ChatGPT answers, a 97% presence rate, which tells you strong, unambiguous brands can still show up reliably even while their exact rank bounces around. Presence is more stable than position. That distinction matters for how you should read any tool's numbers.
Secondary analysis of this research, cited by Passionfruit Labs, suggests you'd need something like 60 to 100 runs of a single prompt to get a statistically meaningful read on visibility. Nobody's rank tracker is running a prompt 100 times a day for every keyword you care about. Budget and API costs alone make that impractical, which means every tool is working from a sample, and different tools take different samples.
Why the underlying model never gives the same answer twice
This isn't a bug anyone can patch. Large language models generate text by sampling from a probability distribution one token at a time. When two or three candidate words are close in probability, which one gets picked can flip from run to run, and that single flip cascades through everything that follows. A piece on temperature and AI determinism (geotoolbox.ai) makes a point that surprised me the first time I read it: even temperature set to zero isn't fully deterministic in production, because floating-point math isn't perfectly associative, and GPU batching on the server side can nudge near-tied probabilities one way or another depending on what else is being processed at the same moment.
On top of that, you as a user have zero visibility into or control over what's actually happening behind your prompt. The exact same text you type might get routed to a different model version, wrapped in a system prompt that changed since yesterday, or grounded in web pages that were freshly retrieved five seconds before your request and five seconds before someone else's identical request. Server load itself is a documented source of variance, according to research cited on alphaXiv: the same prompt can return different results purely because it landed in a busier batch of concurrent requests.
So when two tools show different numbers, they might genuinely both be "right." They just happened to catch the AI mid-flip.
Methodology differences compound the randomness
Even if the models were perfectly consistent, tools would still disagree, because they're not measuring the same thing in the same way. Searchable lays out a useful framework of questions to ask any AI visibility vendor, and it's a good gut check:
- How many times does it run each prompt? A single check once a day is a coin flip, not data.
- Which engines does it query, and through what method? API access and captured UI output can return different answers for the same engine, which is fine if you know it's happening and a real problem if you don't.
- Where do the prompts come from? Synthetic prompts a tool invented versus real search queries people actually typed are not the same distribution. Ahrefs Brand Radar leans on real "People Also Ask" volume data for this reason, arguing that most competitors are "guessing what users might ask."
- How does it define a mention? Being named, being described, being linked, and being cited as a source are four different outcomes, and a blended "visibility score" can rise even when a competitor got the actual recommendation and you only picked up a passing citation.
- Can you see the raw answers, or only a rolled-up number?
Searchable also found a case where two tools tracking the same brand over the same weeks showed ChatGPT naming the brand more often than a comparison engine every single week, a consistent structural gap rather than a fluke. That's a methodology difference, not noise.
What Promptwatch's own citation data reveals about the mismatch
Promptwatch tracks AI search behavior across more than 26 billion analyzed citations and prompts, and its own numbers show exactly how much the ground shifts under any single "rank" measurement.
Sources per answer vary a lot by engine. ChatGPT cites around 5 sources per web-search-enabled response, which Promptwatch calls "the entire citation inventory for a prompt." Google AI Overviews cite roughly double that, about 10. Perplexity is the most predictable engine, landing almost exactly at 10 sources every day, which points to a deliberate retrieval design rather than something that just happens to settle there. Microsoft Copilot is the wild card: its average swung from under 2 to nearly 17 sources per response within a few weeks before dropping back down, which Promptwatch attributes to Microsoft actively re-architecting its retrieval and attribution system. If one tool samples Copilot on a volatile week and another samples it a month later, they'll look like they're measuring different products entirely. More detail on this is in Promptwatch's average sources per response data.
How a prompt is phrased changes the size of the pool you're competing in, too. Promptwatch's data on unique domains per ChatGPT response (covering late September through early October 2026) found that organic, non-branded prompts are the most crowded: 52.5% cite 10 or more domains. Brand-specific prompts narrow sharply, with 21.3% hitting 10+ domains and 13.8% citing three or fewer, triple the organic rate. Two trackers running what looks like "the same keyword" but phrasing the prompt slightly differently, say, an organic query versus a branded one, are quietly sampling from different-sized citation pools. See the full breakdown in Promptwatch's data on unique domains per ChatGPT response.
And citation behavior can change overnight at the platform level, independent of anything on your site. Around the GPT-5.3 rollout on March 4, 2026, average citations per ChatGPT response dropped about 27%, from roughly 6.4 down to 4.7-4.9, across every model variant simultaneously, with no recovery a month later. Promptwatch's own framing of this is the most useful sentence in this entire guide: citation behavior is "a platform-controlled variable that can change overnight, which makes single-snapshot audits unreliable and continuous monitoring essential." Full numbers are in the ChatGPT citation drop report. Before you assume your content lost favor, check whether the date lines up with a model release.
AI Overview-specific failure modes
Beyond randomness and methodology, there's a third category of disagreement: plain detection errors. Omnia's own buyer's guide to AI Overview trackers is refreshingly honest about this, flagging false positives (a tool mistaking a featured snippet or a "People Also Ask" box for an AI Overview) and false negatives (a layout change confusing the parser) as the two failure modes separating functional trackers from unreliable ones. Omnia's FAQ states plainly: "No tool can guarantee 100% accuracy because AI Overviews themselves are non-deterministic."
Keyword.com recommends the obvious but often skipped fix for manual spot-checks: check AI Overviews in incognito mode, logged out of every Google account, with a fixed location and language, because personalization and account history materially change what shows up. SerpApi also notes that AI Overviews were only visible to logged-in Google accounts until relatively recently, which means tools built before official API parsing existed were likely relying on fragile scraping methods that could silently break.
How to actually use these tools without getting burned
| What to check | Why it matters | Red flag |
|---|---|---|
| Runs per prompt | Single-check data is close to a coin flip given <1% repeat-answer odds | Tool reports one snapshot as "your rank" |
| Engines queried and method | API vs. captured UI can return different answers for the same engine | Vendor doesn't disclose collection method |
| Prompt origin | Synthetic prompts vs. real search-volume-backed prompts sample different distributions | No visibility into how prompts were generated |
| Mention definition | Named, described, linked, and cited are different outcomes blended into one score | Only a single "visibility score," no raw answers |
| Trend vs. snapshot | Platform-side citation behavior can shift ~27% overnight after a model update | Dashboard shows one day's number with no history |
A few practical habits help cut through the noise:
Treat every number as a range, not a point. Advanced Web Ranking's advice for classic rank tracking applies even more to AI Overviews: report positions and presence as trends across weeks, not single-day figures. If three different measurements (a tracker's trend, your own manual incognito checks, and a second tool) broadly agree, trust the direction. If they diverge, the divergence itself is diagnostic, usually pointing to a collection or localization quirk rather than an actual visibility loss.
Weight presence over position. Given how unstable exact rank is, whether you show up at all, consistently, across dozens of runs, is a more trustworthy signal than whether you were #1 or #3 in one particular answer.
Ask every vendor the six Searchable questions before you buy. If a sales rep can't tell you how many times they sample a prompt or whether they're hitting an API or a captured UI, that's useful information on its own.
Watch for platform-wide shifts before panicking about your own content. If citations drop across the board the same week a major model update ships, that's not your SEO team's fault.
If you want a platform built around this reality rather than pretending AI answers are deterministic, Promptwatch tracks citation trends, crawler logs, and prompt-level data across ChatGPT, Gemini, Claude, Perplexity, AI Overviews, AI Mode, and more, with continuous monitoring rather than one-off checks, and turns the gaps it finds into prioritized content actions instead of just a score.
For a broader look at the category, the GEO software directory at bestgeosoftware.com is a reasonable place to compare how different platforms handle prompt sampling, engine coverage, and citation reporting before you commit to one number as gospel.