How We Test AI Visibility Tools: Our Methodology for Fair, Repeatable Reviews (2026)

The exact process we use to review AI visibility tools in 2026: fixed prompt sets, repeat runs, raw evidence checks, engine normalization, and a scoring rubric you can reuse to run your own evaluation.

Key takeaways

  • We test every tool against the same prompts, URLs, engines, regions and dates, then verify a sample of findings by hand before trusting any score.
  • Accuracy is not agreement. Two tools can report different scores and both be valid, if each one shows how it measured.
  • Repeat runs matter. Engines drift and sometimes change overnight, so a single snapshot proves very little.
  • We weight measurement fidelity and raw evidence above dashboards, price and polish.
  • You can copy this process with a spreadsheet and two weeks of effort.

Reviews of AI visibility tools are easy to write badly. You sign up, look at the dashboard, screenshot the nicest chart, and call it a day. We did a few of those early on and I still wince at them. This page documents what we do now, so you can judge our reviews and, if you like, reproduce them.

This site is published by 1001 SEO Media, an SEO and GEO agency, and the methodology below is the one we use when evaluating tools for our own client work too. We name that relationship up front because it affects how you should read our recommendations: we use Promptwatch ourselves, and we hold it to the same tests as everyone else.

Favicon of Promptwatch

Promptwatch

Track and optimize your brand's visibility in AI search engines
View more
Screenshot of Promptwatch website

What we are actually testing

An AI visibility tool claims to tell you whether ChatGPT, Gemini, Perplexity, Claude, Google AI Overviews and similar engines mention or cite your brand. That claim has several separate parts, and most bad reviews blur them together:

  • Collection: how the tool gets answers from each engine (real interface, API, or something else).
  • Detection: whether it correctly spots a brand mention, a cited URL, or an absence.
  • Repeatability: whether the same test produces reasonably similar results twice.
  • Evidence: whether you can see the raw prompt, response, engine, date and sources behind each number.
  • Actionability: whether the output tells you what to do next.

A tool that compresses all of this into one unexplained score fails the first test before we run anything. As the VisiblePilot evaluation guide puts it, the useful question is whether you can trace a score back to observable facts.

VisiblePilot guide on evaluating AI visibility tool accuracy with controlled prompts and raw evidence

Step 1: Build a fixed test set

Every tool in a comparison gets the same inputs. No exceptions, even when a vendor offers to "set it up for us."

The prompt set

We write 30 to 50 prompts across three intent groups, which mirrors the approach in the Yotpo and 20North audit guides:

  • Informational: the buyer is learning ("what is X", "how do I choose X").
  • Comparative: the buyer is weighing options ("best X for small teams", "A vs B").
  • Branded: the buyer already knows a name and wants specifics.

For each core question we write three to five natural-language variants. People do not type the same sentence twice, and neither should a test. We also tag each prompt with a persona and a country, because a monitor's location changes the answer.

The site sample

For tools that audit crawlability or content, we pick eight to twelve URLs: homepage, a product page, a topic hub, a long article, a conversion page, a recently published page and at least one JavaScript-heavy page. We record the expected status code, canonical and robots directives before the tool sees anything. If possible, one URL carries a deliberately documented issue so we can check the tool finds it.

The ground truth log

Before any tool runs, we manually query each engine ourselves, two or three times per prompt, and log the answers in a spreadsheet: one row per prompt per engine, consistent column names, date, model where visible, brand mentioned yes or no, cited URLs. This is our reference. It is imperfect, which is the point of the next step.

Step 2: Check how the tool collects answers

This is the single biggest differentiator, and the one vendors explain least. We ask:

  • Does the tool run prompts against the real consumer interface or through an API?
  • Which engines, which countries and languages?
  • How often does each prompt run?
  • Are citations captured as shown to users?

The distinction matters. Proven Cite's published methodology is refreshingly blunt about it: API responses can differ from consumer apps because browsing, personalization, system instructions and model versions differ, so a test is a dated snapshot rather than continuous observation. That is honest, and API-based collection is a legitimate design, but it answers a different question than "what does a buyer see in ChatGPT?"

Proven Cite methodology page explaining API versus consumer app differences and what counts as evidence

Tools that monitor real interfaces get higher marks on measurement fidelity in our rubric. Promptwatch, for example, collects from the actual UIs of ChatGPT, Google AI Overviews, AI Mode, Gemini, Perplexity and Copilot per country and language, and tracks API models alongside to show training-data beliefs. You can read how that works in its data collection academy page. We still verify it, which brings us to the next step.

Step 3: Run repeatability tests

AI answers vary from run to run. A tool that reports a precise visibility percentage from one run per prompt is reporting noise. So we do three things:

  • Run each tool's prompts for at least 14 days to see daily variance.
  • Compare tool output against our manual ground truth on a sample of 20 prompts.
  • Re-run after any major engine change, and log the date.

Why we normalise per engine

Engines behave differently enough that raw comparison is misleading. Promptwatch Data's average sources per response report shows ChatGPT citing about 5 sources per web-search answer, Google AI Overviews about 10, and Perplexity almost exactly 10 every day. Copilot's average swung from under 2 to nearly 17 sources within a few weeks before settling. The report's advice is to judge Copilot on monthly trends, not weekly snapshots, and that is how we treat it too. Perplexity's stability makes it a decent test bench: when your citations move there, your content probably moved, not the engine.

Engines change overnight

The other reason we log dates is that engine behavior shifts abruptly. According to Promptwatch Data, ChatGPT started using the site: operator at scale on August 8, 2026, jumping from roughly 0.4% to 17% of fanout queries (report), and reddit.com's share of ChatGPT citations fell from about 4% to 0.5% on August 14, 2026 (report). Earlier, average citations per response dropped around the GPT-5.3 rollout on March 4, 2026 (report). Fanout behavior moves too: the ChatGPT query fanouts report shows average searches per response falling from about 2.15 in December to about 1.84 by March.

The consequence for a reviewer is simple. If a tool's numbers jump on one day, the first suspect is the engine, not the brand and not the tool. A good product lets you annotate or at least see the change. We mark that as a plus.

Step 4: Verify the evidence trail

For every tool we pick 15 to 20 reported findings and try to confirm them by hand. The questions:

  • Can I open the underlying response, with its prompt, engine, date and location?
  • Does it distinguish a brand mention, a linked domain and an absence?
  • Does a cited URL actually appear in the sources list, or was it only mentioned in prose?
  • Are false positives handled? A brand name that is also a common word is the classic trap.

Proven Cite makes a useful point here: a link in generated output is not proof the provider retrieved or relied on that page. We keep mentions and citations separate in our notes, the way Promptwatch's citations vs mentions guide frames it. Tools that let you write plain-language rules for what counts as a mention (Promptwatch calls these Brand Book analysis instructions) score better on false-positive handling.

Step 5: Judge actionability and workflow

A tracker that reports a score and stops is a dashboard. We test what happens after the number:

  • Does it tell you which pages AI cites, and where gaps are?
  • Does it show crawler access, so you can separate an access problem from a content problem?
  • Does it connect to traffic, so you can see whether visibility ever becomes visits?
  • Can a team assign and track the work?

This is where categories separate. Pure trackers like Otterly.AI and Peec AI are simple and affordable. Suites like Semrush add AI tracking to existing SEO workflows. Platforms such as Promptwatch go further into crawler logs, content generation and publishing. We do not treat "more features" as automatically better; a small team that only needs a weekly mention report may be happier with a lean tool.

Favicon of Otterly.AI

Otterly.AI

Affordable AI visibility monitoring
View more
Screenshot of Otterly.AI website
Favicon of Peec AI

Peec AI

Multi-language AI visibility tracking
View more
Screenshot of Peec AI website
Favicon of Semrush One

Semrush One

Unified SEO and AI visibility platform
View more

Step 6: Score it

We use a weighted rubric. It is adapted from published frameworks (Percepture's CLEAR editorial weighting is one reference point) but tuned for what we care about:

CriterionWeightWhat we look for
Measurement fidelity20%Real-interface or documented API collection, citations as users see them
Engine coverage15%Number of engines, countries and languages
Prompt methodology15%Volumes, intent tagging, personas, fan-outs, custom prompts
Evidence and repeatability15%Raw responses, history, run frequency, variance visible
Actionability15%Gaps, recommendations, content and crawler insight
Competitive intelligence5%Competitor discovery, share of voice
Traffic and attribution5%Visitor analytics, Search Console links
Price and value10%Cost per tracked response, not sticker price

Cost per response matters more than monthly price. A $95 plan with 6,000 responses and a cheaper plan with 1,000 are not comparable at face value. We calculate both and show the maths in each review.

Step 7: What we publish, and what we refuse to

Every review notes the test dates, engines covered, prompt count, and which plan we used. We publish negative results. We label anything we could not test as "not tested" instead of guessing, a habit borrowed from Proven Cite's methodology, which does the same.

We do not:

  • Rank tools by dashboard looks.
  • Treat a vendor's own benchmark as evidence.
  • Claim a tool caused a visibility change when many things moved at once.
  • Reuse results older than a quarter without re-checking.

That last one follows the quarterly cadence recommended in the 20North audit guide: run a full baseline once, then repeat every quarter. Given the engine shifts above, we sometimes re-run sooner.

20North guide on running an AI visibility audit step by step

Comparison: what we check by tool type

Tool typeTypical collectionOur main concernExamples
Prompt trackersUI or API, variesSmall prompt counts, thin evidenceOtterly.AI, Peec AI, Rankscale
SEO suite add-onsOften fixed prompt setsShallow AI-specific dataSemrush, Ahrefs Brand Radar
Enterprise GEO platformsReal UI, large panelsPrice, lock-inProfound, Scrunch, Brandlight
End-to-end platformsReal UI plus logs and contentBreadth versus depth in each areaPromptwatch
One-off gradersSingle homepage scanOne run is not a baselineHubSpot AEO Grader, FalconRank

For wider lists by category, the GEO software directory is a good place to browse alternatives we have not covered yet.

Favicon of Profound

Profound

Track and optimize your brand's visibility across AI search engines
View more
Screenshot of Profound website
Favicon of Scrunch

Scrunch

AI visibility tracking for influencer marketing
View more
Screenshot of Scrunch website
Favicon of Ahrefs Brand Radar

Ahrefs Brand Radar

Brand monitoring in AI search results
View more
Screenshot of Ahrefs Brand Radar website

Where this method is weak

I would rather tell you than have you find out.

  • Our manual ground truth is itself a sample. We cannot see private conversations or global prompt volumes, and neither can anyone else.
  • Two weeks of daily data catches variance but not seasonality.
  • Vendors ship changes constantly. A review can be stale in a month, which is why every one carries its test dates.
  • We cannot fully separate a tool's quality from the plan we tested. Higher tiers often add faster tracking or local targeting.

Run it yourself

You do not need our rubric to avoid being misled. A minimal version:

  • Write 20 buyer questions with three variants each.
  • Query each engine manually, two or three times, and log it in a spreadsheet.
  • Trial two or three tools with the same prompts for the same dates.
  • Open the raw response behind ten numbers in each tool and check them.
  • Ask each vendor to walk through your own brand, not a demo account.

Most vendors, Promptwatch included, offer a 7-day free trial on monthly plans, plus a free AI visibility report if you only want a quick baseline. Confirm current pricing at promptwatch.com/pricing before you commit, since plans change.

If you would rather hand the evaluation to someone else, that is a service 1001 SEO Media provides for brands choosing a platform, and the contact is [email protected]. Otherwise, copy the steps above. They work.

Share:

© 2026 AI Search Tools · Best AI search tools and platforms · RSS

AI Search Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.