Key takeaways
- AI citation behavior changes overnight (ChatGPT's citations per response dropped ~27% around the GPT-5.3 rollout), so any audit based on a single snapshot is close to useless. You need to know how a vendor samples, not just what number they show you.
- Sample size matters more than most buyers realize: a single prompt run once needs roughly 97 runs to get a ±10-point confidence interval, and 385 runs for ±5 points. Ask any vendor exactly how many runs per prompt per engine they're doing.
- Only about 12% of URLs cited by AI assistants also rank in Google's top 10, and ChatGPT's sources overlap with Google's by roughly 39%. Your existing rank tracker cannot substitute for AI visibility data.
- Citation counts, source counts, and even which platforms get cited (Reddit's ChatGPT citation share fell from ~3.8% to 0.5% in days) all move independently by engine. A platform that lumps every AI engine into one score is hiding the signal you actually need.
- Before paying $95 to $5,000+ a month for a GEO platform, run this 20-point audit yourself, on your own brand, so you know what a legitimate baseline looks like and can hold any vendor's dashboard accountable to it.
Why this audit exists
Every week there's a new AI visibility tool launching with a dashboard, a "visibility score," and a demo that looks convincing. There are now more than 40 tools in this category, and one analysis I keep coming back to points out that the core mechanic behind most of them, schedule some prompts, store the responses, draw a chart, is simple enough that a single developer could build it in a few weeks. That's not a knock on the category. It's a warning that the differentiation isn't in the dashboard. It's in the methodology sitting underneath it, and most buyers never check that methodology before they sign a 12-month contract.
This guide is the audit I'd run myself before buying anything. It's organized into four groups: statistical reliability, citation and engine mechanics, methodology red flags, and vendor coverage. Do this before you demo a single platform. It takes an afternoon and it will change which questions you ask on the sales call.
Group 1: statistical reliability (data points 1-6)
1. How many times does the vendor run each prompt?
LLM answers are probabilistic. Run the identical prompt twice and you can get different brand lists both times. A SparkToro study running 12 prompts across ChatGPT, Claude, and Google AI Overviews for categories like chef's knives and cancer hospitals found ChatGPT and AI Overviews produced the same brand list less than 1% of the time on repeated identical queries. Claude was marginally better at 2-3%. If a vendor runs each prompt once per day and calls that your "visibility score," you're looking at noise dressed up as data.
2. What sample size gets you a usable confidence interval?
A visibility rate near 25% measured over 72 answers carries roughly a ±10 percentage point margin of error at 95% confidence. Tightening that to ±5 points needs around 300 answers. For a single prompt on a single model, that's 97 runs for ±10pp and 385 runs for ±5pp. Detecting a real move from 20% to 30% visibility requires around 294 answers per measurement period. Ask the vendor to show you this math for your account, not their marketing page.
3. Does the vendor group prompts into topics, or track them individually?
Grouping 15-20 related prompts into a topic and averaging across them cuts the margin of error dramatically, from roughly ±16 points on a single prompt to under ±4 points weekly once you're running around 100 prompts across a few models and topics. If a platform sells you 10 or 15 prompts total and treats each as its own independent metric, you're paying for statistical noise with a nice UI wrapped around it.
4. How stable are citations between two runs minutes apart?
This one surprised me. A paired-run test where the same model answered the same 20 prompts minutes apart found only about 40% of cited domains repeated between the two runs (a mean Jaccard similarity of 0.40). That means a single "citation snapshot" telling you which domains got cited today is barely more reliable than a coin flip on domain-level detail, even with zero time passing between measurements.
5. Does the vendor tell you how many prompts and runs are included in your plan, in plain numbers?
Walk through actual plan tiers: Otterly's entry plan gives you 15 prompts, Profound's starter plan gives 50 prompts on ChatGPT only, Scrunch's starter tier gives 350 custom prompts across 3 users. None of these numbers mean anything until you know the run frequency per prompt. A 50-prompt plan sampled once daily across 6 engines gives you 2,100 answers a week, which is enough for a real week-over-week read. The same setup measured over a single day gives you roughly 300 answers, too thin for a day-over-day claim, even though the vendor's dashboard will happily draw you a daily line chart anyway.
6. Can you export raw prompts and responses, not just the aggregated score?
If you can't see the underlying prompt text and the raw model response, you can't debug an anomaly or verify the vendor's methodology claims at all. This is a binary yes/no question worth asking directly on the sales call.
Group 2: citation and engine mechanics (data points 7-14)
7. How many sources does each engine actually cite per response?
This varies enormously and it changes the size of the prize you're fighting for. ChatGPT cites around 5 sources per web-search-enabled response. Google AI Overviews cites roughly double that, around 10. Perplexity is remarkably consistent at almost exactly 10 sources per answer, day after day. Microsoft Copilot is the outlier: its average has swung from under 2 sources per response to nearly 17 within a few weeks, which tells you Microsoft is still actively rebuilding its retrieval and attribution system underneath the product. Source: Promptwatch's data on average sources per response.
8. Has the platform's citation baseline shifted recently, independent of anything you did?
Around the GPT-5.3 rollout on March 4, 2026, average citations per ChatGPT response dropped from roughly 6.4 the week before to 4.7-4.9 by late March, a nearly 27% drop, across every model variant simultaneously, with no recovery a month later. If your visibility score dropped that week, it wasn't your content. It was the platform. Ask any vendor pitching you a GEO tool whether they'd have flagged this as a platform-wide event rather than a brand-specific problem. Source: Promptwatch's data on the ChatGPT citation drop.
9. Does the platform separate engines instead of blending them into one score?
Reddit's share of ChatGPT Search citations held around 3.8% for weeks, then collapsed to under 1% within days in mid-August 2026, an 86% relative drop. Over the same window, Google AI Overviews and AI Mode only drifted down gradually, no cliff at all. If a vendor's dashboard shows you one blended "AI visibility" number across engines, you'd never catch a shift like this. Source: Promptwatch's data on Reddit citations dropping in ChatGPT.
10. What content type is actually winning citations in your category right now?
In July 2026, product pages became ChatGPT's most-cited content type at roughly a third of all daily citations, nearly double their March share. Listicles, how-tos, and social posts were all growing too, mostly at the expense of plain product and landing pages elsewhere in the mix. If you're optimizing blog posts while your competitors' product pages are eating citation share, you're solving the wrong problem. Source: Promptwatch's citation types data for July 2026.
11. Does the vendor track query fan-outs, not just final answers?
ChatGPT breaks a single prompt into multiple separate web searches behind the scenes, and this behavior changes fast. On August 8, 2026, ChatGPT Search started using the site: operator at scale, jumping from under 0.4% to nearly 17% of all fan-out queries overnight, and the number of queries per response nearly doubled the same day. Practically, this means ChatGPT is now deliberately targeting specific domains for information rather than just crawling the open web generically, so thin or unindexed pages on your own site now cost you AI answers directly. Source: Promptwatch's data on the ChatGPT site: operator shift.
12. Can the tool tell you which AI crawlers can actually reach your site?
OpenAI accounted for nearly 80% of verified AI crawler requests in early September 2026, down from about 95% in June, a real mix shift toward Anthropic, Google, Perplexity, and Mistral crawlers within three months. Compare your own server logs against this mix. If a crawler operator shows a large share in aggregate data but near zero hits on your own domain, you're likely blocked or unreachable for that provider, and no amount of content optimization fixes that. Source: Promptwatch's AI crawler traffic data.
13. Does the platform measure with live retrieval on, or off?
A brand can vanish entirely in a "retrieval off" test simply because it was never in the model's training data, which tells you nothing about real-world AI search visibility. Conversely, a rising score with retrieval on might just mean you temporarily won a search result for one query, not that the model has learned anything durable about your brand. Ask the vendor which condition every metric on their dashboard reflects. If they can't answer clearly, that's a red flag on its own.
14. Does the tool flag hallucinated claims about your brand, not just mention counts?
AI systems can and do invent pricing, features, or policies that don't exist, the kind of thing that got Air Canada in real trouble when its own chatbot invented a refund policy. A mention count tells you nothing about whether the mention is accurate. Ask whether the platform monitors for factual drift in what models say about you, separate from whether they say anything at all.
Group 3: methodology and vendor due diligence (data points 15-20)
15. Where do the prompts come from, real query data or a guess?
This is probably the single biggest quality signal in the category. Ahrefs Brand Radar builds its prompt set from real "People Also Ask" queries with measurable search volume, over 243 million of them. Most competitors instead construct or guess prompts that sound plausible. Ask directly: are these prompts sourced from real search demand, or written by someone on the product team trying to imagine what a customer might ask?
16. How many prompts per topic is the minimum before the noise drowns the signal?
Multiple independent analyses converge on the same rule of thumb: fewer than 10-20 prompts per topic creates statistical noise too high to draw conclusions from. If a plan tier caps you at 10 or 15 prompts total across your entire brand, you don't have enough data to say anything meaningful about even one topic, let alone your whole category.
17. Does the tool separate absolute citation rate from relative share of voice?
A 40% citation rate means nothing without context, on its own it doesn't tell you if you're winning or losing. What matters is your citation rate compared to competitors, your position within the answer (cited first versus buried in paragraph four), and whether the citation includes a link or just a passing brand mention. Rough bands worth knowing: 60%+ suggests strong authority, 30-60% is competitive but not dominant, under 30% usually means the content or the targeted queries need work, and 0% usually means a trust or content-adequacy problem, not bad luck.
18. Can you see the tool's engine coverage tier by tier, not just the marketing page?
Vendor plans routinely gate engine coverage behind pricing tiers in ways that aren't obvious until you're already a customer. Profound's entry plan tracks ChatGPT only; you need the Growth tier to add Perplexity and Google AI Overviews, and Enterprise to unlock the full roster including Gemini, Copilot, Claude, Grok, and DeepSeek. Peec AI caps self-serve tiers at 3 engines regardless of price point unless you're on Enterprise. Ask for the exact engine list on the specific plan you're about to buy, not the plan featured in the case study.
19. Does the platform go beyond monitoring into actual fixes?
Most of the category stops at telling you a number went up or down. A smaller set of platforms, including Promptwatch, pair the monitoring with content gap analysis, automated content generation, and CMS publishing that actually closes the loop from "you're invisible on this prompt" to a published page addressing it. Promptwatch's crawler log data (the same AI crawler traffic dataset referenced above) comes from its own dataset of over 4.5 billion citations, clicks, and prompts analyzed across 1,840+ customers, which is a different scale of evidence than a tool that only checks a handful of prompts a day.
20. What does a quarter of ongoing monitoring actually cost, all-in?
Pricing across this category ranges from roughly $29/mo (Otterly's Lite plan, 15 prompts) up to $5,000+/mo for top-end enterprise setups, with a category average sitting around $337/mo across roughly 20 tools surveyed. But the sticker price rarely includes everything: Ahrefs Brand Radar's full 6-index coverage plus base plan runs $527-$1,148/mo all-in, and Semrush dropped its free tier entirely in August 2026 after Adobe's $1.9 billion acquisition closed, moving to a paid-only entry point. Get the fully-loaded monthly number, not the homepage number, before you compare vendors.
Putting it together: a comparison snapshot
| Platform | Entry price | Prompt volume (entry) | Engines at entry | Goes beyond monitoring |
|---|---|---|---|---|
| Promptwatch | $95/mo (Essential) | 50 prompts, 6,000 responses | All major LLMs from entry tier | Yes — content agents, CMS publishing, crawler logs |
| Profound | ~$99/mo | 50 prompts | ChatGPT only | Limited (FactCheck remediation add-on) |
| Ahrefs Brand Radar | $50-$250/mo per index | Varies by index | Depends on index selected | No, monitoring-focused |
| Otterly.AI | $29/mo | 15 prompts | 4 core engines | No, monitoring-focused |
| Scrunch AI | $250-$300/mo | 350 custom prompts | 7 platforms | Partial (agent-traffic attribution) |
| AthenaHQ | ~$95-$245/mo | Credit-based | Multiple, tier-gated | Partial (predictive citation engine) |
Running the audit on yourself first
Before any vendor call, spend an afternoon doing this manually. Pick 15-20 real prompts your customers would plausibly type, run each one 5-10 times across ChatGPT, Perplexity, and Google AI Overviews, and log which domains get cited, in what position, and whether the mention is accurate. That gives you a rough baseline and, more importantly, a feel for how much variance exists run to run. When a vendor later shows you a smooth trend line with no variance at all, you'll know to ask why their data looks cleaner than reality actually is.
If you want to browse the wider field before committing, the GEO software directory at bestgeosoftware.com covers a broader set of platforms side by side, and ai-rank-tools.com focuses specifically on rank-tracking-style tools for AI search if that's the narrower job you're trying to solve.
The honest conclusion from all 20 of these checks is that the dashboard is the easy part. What separates a tool worth paying for from one that's quietly wasting your budget is whether the number on the screen is statistically real, tied to a specific engine and content type, and connected to something you can actually go fix.