Key takeaways
- A tool is paying off only if it changes a decision or a business number. Mention counts alone do not qualify.
- Measure in three layers: visibility (are we in the answers), behavior (are AI systems reading and citing our pages), and outcomes (traffic, conversions, pipeline).
- Set a baseline before you judge anything. Without a day-zero snapshot, every chart looks like progress.
- Citation share decays. Promptwatch Data shows news pages losing half their ChatGPT citations within 36 days, so re-measure on a schedule.
- Review at 30, 60, and 90 days, and cancel or consolidate tools that only produce dashboards.
Why most AI search tool reviews fail
I have seen the same pattern a lot. A team signs up for an AI visibility tool, sets up 50 prompts, and watches a line chart for a month. The line goes up a bit. Everyone feels fine. Nobody can say what the tool actually changed.
The problem is that the tools are cheap to buy and easy to look at, so the question "is this working?" never gets asked properly. Third-party claims make it worse. You will hear that 58% of marketers say AI search traffic converts better than regular organic (that figure comes from a Semrush-sponsored Exposure Ninja video, so treat it as a vendor-adjacent claim, not a benchmark). You will also hear that Gartner predicted a 25% drop in traditional search volume by 2026. That was a 2024 forecast, not a measured outcome.
Your own data beats both. This guide shows how to build it.
Step 1: Decide what the tool is supposed to do
AI search tools fall into roughly four jobs. Be honest about which one you bought, because each has a different success test.
| Job | Typical tools | Success looks like |
|---|---|---|
| Monitoring (are we mentioned?) | Otterly.AI, Peec AI, Rankscale | Stable, trustworthy visibility and share-of-voice numbers |
| Diagnosis (why or why not?) | Crawler logs, site audits, citation analytics | You find and fix specific blockers |
| Action (fix it for me) | Content agents, action boards | Published pages that get cited |
| Attribution (did it make money?) | Visitor analytics, CRM and attribution tools | AI-referred sessions tied to leads and revenue |
A pure monitoring tool can be worth it, but only if you do something with the output. If your team bought a tracker and expected revenue, the tool is not the failure. The expectation was.
Step 2: Take a baseline before you change anything
Before you publish a single optimized page, record:
- Visibility score or share of voice for a fixed set of prompts, per engine
- Position in the answer when you are mentioned
- Number of your own pages cited (self-citation rate), not just brand mentions
- Sentiment of the answers about you
- AI-referred sessions and conversions from analytics, even if the number is tiny
- Branded organic traffic, which the Exposure Ninja video flags as a useful companion signal
- Which AI crawlers hit your site, and what errors they get
Track each engine separately. Promptwatch Data's average sources per response report shows why: ChatGPT cites about 5 sources per web-search answer, while Google AI Overviews and Perplexity cite about 10. A citation in ChatGPT is scarcer, so blending engines into one number hides what is happening.
A fixed prompt set also matters. If you keep swapping prompts, you are measuring your prompt list, not your visibility.
Promptwatch can hold this baseline across real ChatGPT, AI Overviews, Gemini, and Perplexity interfaces, and so can several others in the comparison below.
Step 3: Measure in three layers
Layer 1: Visibility
This is the layer every tool covers. Read the metrics together, because each one points at a different problem:
- Low share of voice usually means a coverage problem. You are missing from the topics.
- Late position in the answer means a prominence problem.
- Mentions without self-citations suggest a content or crawlability problem. AI knows your name but does not use your pages.
- Weak sentiment is a reputation problem, often traced to third-party sources.
If your tool reports one blended score and nothing else, you cannot diagnose anything. That is a real limitation worth weighing at renewal.
Layer 2: Behavior
This layer is where most teams have a blind spot. Visibility is the outcome. Crawling and citing are the mechanism.
Check your server or CDN logs for AI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others. Promptwatch Data's AI crawler traffic report (IP-verified requests, June 8 to Sept 6, 2026) shows OpenAI at 79.8% of verified requests in the week of Aug 31 to Sept 6, down from 94.8% in early June. The practical test: compare that mix against your own logs. If a provider is large in the market data but near zero on your site, you may be blocked by robots.txt, a CDN rule, or a WAF.
Tools with built-in crawler logs make this easy. Alternatives include DarkVisitors for agent and bot tracking, or a manual log review.
Also run a site audit. Broken links, orphaned pages, and missing sitemap entries keep AI from reading your best content. Screaming Frog and Sitebulb work for this if your visibility tool lacks one.
Layer 3: Outcomes
This is the layer that answers the CFO. You need AI-referred traffic and conversions in your own analytics.
- Set up GA4 conversion events first. The Exposure Ninja video makes this point well: conversion events, not traffic, should anchor the measurement strategy.
- Create a channel group or segment for AI referrers (ChatGPT, Perplexity, Gemini, Claude, Copilot and so on).
- Compare conversion rate, lead-to-opportunity rate, and revenue per lead for AI traffic against organic.
- Watch branded organic search. Buyers who meet you in an AI answer often search your name later, and that visit never shows as an AI referral.
For B2B, push this into your CRM. Attribution platforms like Dreamdata and HockeyStack can stitch AI-referred sessions to pipeline, and Cometly ties attribution to AI visibility directly.
Be realistic. AI referral volume is small for most sites and a lot of influence is invisible (the buyer reads the answer, never clicks, and shows up later as direct or branded). So report AI-referred conversions as a floor, not a total.
Step 4: Pick the right metrics for each tool type
| Question | Metric | Where it comes from |
|---|---|---|
| Are we showing up? | Visibility score, share of voice, position | AI visibility tracker |
| Are our pages used? | Self-citation rate, citation rank | Citation analytics |
| Can AI reach us? | Crawler hits, crawl errors, audit score | Crawler logs, site audit |
| Is it driving visits? | AI-referred sessions by engine | Visitor analytics, GA4 |
| Is it driving money? | Conversion rate, pipeline, revenue per lead | GA4 plus CRM |
| Is the work shipping? | Pages published, pages cited within 30 days | Content workflow |
If a tool cannot feed at least two rows of this table, ask what you are paying for.
Step 5: Account for decay
AI citations are perishable, and this changes how you judge a tool over a quarter. Promptwatch Data's content half-life report tracked every page ChatGPT cited on July 6, 2026 for 90 days:
- After 30 days, ChatGPT still cited 78% of landing pages and 76% of docs.
- Product pages, reviews, and how-tos sat around 55% after 60 days, reaching half-life at 74 to 77 days.
- News was worst: half of cited news pages were gone after 36 days.
So a visibility bump from a press-style piece is a short burst, and a bump from a landing page or docs lasts. When you evaluate a tool, split your wins by content type. A dashboard that celebrates a spike in news citations in week two is telling you less than it seems to.
It also means a good tool should help you refresh decaying pages. Re-measure on a schedule, and treat a refresh as part of the cost of the channel.
Step 6: Compare tools on whether they close the loop
The catalog of AI visibility tools is huge now. A rough sorting by what they actually do beyond tracking:
| Tool | Mostly monitoring | Crawler or traffic data | Content or action workflow |
|---|---|---|---|
| Promptwatch | No | Yes, IP-verified crawler logs and visitor analytics | Yes, Content Agent, Action Board, CMS publishing |
| Profound | Mostly | Limited | Limited |
| Otterly.AI | Yes | No | No |
| Peec AI | Yes | No | No |
| Semrush One | Partly | Search data, shallow AI data | Some content tools |
| Ahrefs Brand Radar | Yes | No | No |
| AthenaHQ | Mostly | Limited | Some |
This is a general positioning summary, so confirm current features and pricing on each vendor's site before you buy.
Where Promptwatch differs is that it is built to answer "why" and "what next" as well as "were we mentioned." It combines prompt tracking with crawler logs, visitor analytics, and an Action Board of prioritized fixes, and its Content Agent can publish to WordPress, Webflow, or Framer, after which it tracks whether each page gets crawled, cited, and clicked. That last step is the one most measurement setups lack: tying a specific published page to a specific change in citations. Plans start at $95/month for Essential, and you can confirm current pricing at promptwatch.com/pricing.
For wider browsing, the GEO software directory lists many more options by category.
Step 7: Run a 30/60/90-day review
Day 30: is the data trustworthy?
- Do repeat runs of the same prompts give stable answers, within normal noise?
- Are you tracking real interfaces or only API outputs? User-facing answers can differ.
- Is the crawler and traffic data connected and showing real numbers?
- Did the tool surface at least three concrete issues or opportunities?
If the answer to the last question is no, the tool is a screen, not an analyst.
Day 60: did the work ship?
- How many fixes did the team make (robots.txt, schema, content gaps, refreshed pages)?
- How many new pages were published against tracked prompts?
- Have those pages been crawled yet? A page AI has never fetched cannot be cited.
Day 90: did the numbers move?
- Share of voice and self-citation rate on your fixed prompt set, versus baseline
- AI-referred sessions and conversions versus baseline
- Branded organic trend
- Cost per tracked prompt and cost per cited page versus what you paid
A simple scorecard works. Give each metric a baseline, a day-90 value, and a verdict: improved, flat, or worse. Then decide whether to keep, change, or cut the tool.
Step 8: Calculate a defensible ROI
I would avoid inventing a precise ROI figure early. You do not have benchmarks to compare against, and published ones are thin. Use a simple structure instead:
- Direct return: AI-referred conversions times average deal or order value, using GA4 and CRM data
- Assisted return: pipeline from accounts that touched an AI referral, via your attribution tool
- Avoided cost: hours saved on manual prompt checking, reporting, and content briefs
- Total cost: software, seats, content production, and the team time to act on findings
Present direct return alone as the conservative case, and add assisted return as upside. Finance teams trust a number they can trace.
If you lack in-house capacity to run this, an agency can set it up. 1001 SEO Media works on AI search and GEO measurement for brands that want senior help without a long contract, and it uses Promptwatch for AI visibility work.
Common mistakes
- Changing the prompt set mid-quarter, which wrecks comparisons
- Judging weekly noise. Copilot's average source count swung from under 2 to nearly 17 within a few weeks in Promptwatch's data, so use monthly trends there.
- Counting mentions as wins when none of them cite your own pages
- Ignoring crawlers, then wondering why optimized content never appears
- Expecting clicks to capture all the value of AI answers
- Keeping three overlapping monitoring tools because nobody owns the review
A short decision rule
Keep a tool if, after 90 days, it did at least two of these: found a fixable problem you did not know about, produced published content or fixes that moved a tracked metric, or connected AI visibility to sessions or revenue. If it only produced charts, downgrade it to the cheapest plan or replace it with something that closes the loop.
For more category browsing, the AI rank tracking directory is a good place to compare trackers side by side before you renew.