Key takeaways
- The same prompt run through a chatbot interface and through its API often returns different sources and different brands. Promptwatch's August 2026 test found only 25.6% source overlap on commercial prompts.
- API polling tends to over-report brand counts and under-report citations, so share of voice from an API can look nothing like what buyers see.
- API tracking is still right for models with no consumer search product (DeepSeek, Mistral) and for measuring what a model believes from training data.
- Sampling frequency matters as much as the collection method. One run per week is a poll, not a measurement.
- Ask every vendor four questions: which route per engine, from which location, is search enabled on API calls, and how often is each prompt run.
Why this question matters now
Most teams pick a prompt tracker by looking at the dashboard, the price, and the number of prompts. Almost nobody asks how the answers were collected. That is a problem, because the collection method changes the numbers more than most of the other settings you will spend time on.
There are two routes. UI collection opens the real consumer interface (ChatGPT, Perplexity, Gemini, AI Mode, AI Overviews), submits the prompt, and parses the rendered answer, brand mentions, and citation links. API polling sends the prompt to a model endpoint and reads the text that comes back. The second is cheaper, faster, and far easier to build. The first is what your customers actually see.
Promptwatch's glossary entry on UI vs API tracking puts it plainly: for the same prompt, the two routes often return different answers, different sources, and different brand recommendations. So the choice changes what your visibility number means.
What actually differs between the two routes
A consumer interface is not just a model. It wraps the model in retrieval, ranking, safety filtering, personalization, and answer shortening. An API call gives you the raw model without that logic. Think of the interface as the finished dish and the API as an ingredient. Neil Patel made a similar point about prompt tracking in general: it works best as directional evidence, and the real UI is the finished product users see.

Sources and citations
Promptwatch ran commercial prompts through the ChatGPT interface and OpenAI's API (web search off) on the same day in the same locale. Results from that August 2026 test:
- The two routes shared only 25.6% of their sources.
- 84% of interface answers had sources, against 26% of API answers.
- The interface produced 392 citations across 149 domains. The API produced 170 across 67.
- 87% of the API's "citations" pointed at homepages typed from memory, not retrieved pages.
- In one answer the API recommended a company that does not exist.
That last point is the one I would show a skeptical stakeholder. A tracker that counts a hallucinated homepage as a citation is measuring the model's imagination.
Turning web search on in the API narrows the gap but does not close it. The same glossary notes two 2026 vendor studies where ChatGPT's API cited 3.1 sources per answer against 12.1 in the interface, with 4.8% domain overlap. That fits what Promptwatch Data shows more broadly: almost half of ChatGPT responses cite 10 or more unique domains (Sep 28 to Oct 4, 2026, see the unique domains report). A thin API answer with three sources is a different object.
Brands named per answer
Surfer ran 1,000 prompts across ChatGPT, Perplexity, Gemini, AI Mode, and AI Overviews, once by scraping the UI and once through an API. One caveat: Surfer sells a UI-scraping product, so it has an interest in the result, and its AI Overviews "API" is borrowed from AI Mode because no AI Overviews API exists. Its findings:
- Jaccard overlap of brands named was 15.5% to 23.8% as returned, and 21.3% to 31.6% after canonicalizing brand names.
- ChatGPT's API named 13.8 brands per answer against 7.9 in the UI.
- ChatGPT's API wrote about 2.05x the words of the UI (863 vs 421). Gemini and AI Mode were close to parity, and Perplexity was reversed, with the UI slightly longer.
- Surfer's takeaway: API-based share of voice has roughly a 25% chance of matching what buyers see, and it over-reports how many brands make the cut.
Treat the exact percentages with some care, since the vendor has a stake. The direction matches the independent Promptwatch test, though, and the mechanism is easy to believe. A longer, unfiltered answer names more brands, which dilutes your share of voice in ways a real buyer would never see.
Things that only exist in the interface
Shopping cards, ads, image carousels, and local panels are rendered UI elements. An API never returns them. If you sell products, or care about sponsored placements in AI answers, API polling is blind to a whole category of visibility.
Query fan-outs are another one. The sub-searches an engine runs behind a response are visible only through the interface. Promptwatch's query fan-out data shows how the average number of web searches per ChatGPT response and the length of those queries shifted between December and April. Those shifts also change which pages get cited, and API tracking cannot see any of it. Promptwatch also found ChatGPT began using the site: operator at scale on August 8, 2026, jumping from about 0.4% to about 17% of fanout queries (report). That is a retrieval behavior change you would only notice from the interface side.
Location and silent changes
An interface session from Germany sees German results. An API call has no browsing context unless the vendor builds one in. And the interface's model, system instructions, and search stack can change without any API announcement. So your interface-based numbers can move for reasons that never show up in a changelog.
The sampling problem sits on top of this
Switching to UI data does not make tracking deterministic. LLMs are probabilistic. In a talk at iPullRank's SEO Week, Aimee Jurenka cited ACL and NeurIPS research showing that identical prompts produce different outputs, rankings invert between runs, and even hardware configuration adds variance. Her argument: single-prompt tracking over time measures noise, and the fix is distributional, topic-level sampling with confidence intervals.

Kevin Indig made a related point in Growth Memo in June 2026: AI visibility is measurable, but the mistake is running it like a rank tracker. The article is paywalled, so I only have the subtitle, but it matches the rest of the evidence.
Chris Green framed the tradeoff on LinkedIn: more accurate data is better for a low number of prompts, and a more complete view is better for larger pieces of analysis. And a writeup on how visibility tools collect data makes the frequency point well: one run per week is a poll, not a measurement, and good tools smooth run-level noise and flag model-update shifts instead of presenting them as ranking changes.
So you need both things: the right collection route and enough repeated runs. Getting one without the other still leaves you with a shaky number.
Side by side: how your numbers change
| Metric | Real UI collection | API polling (search off) | API polling (search on) |
|---|---|---|---|
| Share of sources vs UI | Baseline | About 25.6% overlap in Promptwatch's test | Better, but domain overlap as low as 4.8% in one vendor study |
| Answers with sources | 84% in Promptwatch's test | 26% | Higher, still fewer sources per answer (3.1 vs 12.1 in one study) |
| Brands per answer (ChatGPT) | 7.9 in Surfer's study | 13.8 | Not measured in these sources |
| Citation quality | Retrieved pages | Often homepages from memory | Retrieved, but narrower |
| Shopping, ads, local panels | Visible | Invisible | Invisible |
| Fan-out queries | Visible | Invisible | Mostly invisible |
| Location realism | Real session in target country | None by default | Depends on vendor build |
| Cost and speed | Higher cost, slower | Cheap and fast | Cheap and fast |
When API tracking is the right call
I don't want to make this sound like APIs are useless. They have three legitimate uses:
- Cost and speed at scale, when you need a rough read on thousands of prompts.
- Models with no consumer search product, such as DeepSeek and Mistral, where the API is the product.
- Measuring what a model believes about your brand from parametric knowledge, with no retrieval involved. That is a different question from "will I show up when a buyer searches," and it is worth asking separately.
The hybrid setup Promptwatch recommends is sensible: collect ChatGPT, Perplexity, Gemini, Copilot, and AI Overviews from their real interfaces in the target country, and call DeepSeek and Mistral through their APIs. Promptwatch also tracks API models alongside the interface data to show the training-data view.
How the major tools collect data
Here is the awkward part. My research did not verify which vendors in the catalog use UI collection and which use API calls, and vendors change this over time. Don't trust a table that claims otherwise, including one from me. What I can say from the information available:
Promptwatch states that it monitors the real interfaces of ChatGPT, Google AI Overviews, Google AI Mode, Microsoft Copilot, Gemini, Perplexity, Amazon Alexa, and Meta AI by country and language, with API models tracked alongside. It also layers IP-verified AI crawler logs and visitor analytics on top, so you can check tracked visibility against what bots read and what traffic arrives. Its Visibility Score is built from many responses per monitor, with prompts running daily on every plan and faster on Business and above.

Surfer SEO, a UI-scraping vendor, published the comparison study above, so its position is clear.

For the rest, ask. These are common names you may be evaluating:


Cloro is worth noting for a different reason: it describes itself as a tool with a mention-rate API, which tells you something about how the data is consumed, though not how it is collected.
For more options beyond this list, the GEO software directory and the AI rank tracking tools directory cover many more vendors.
Questions to ask any vendor
- Which route does each engine use, UI or API?
- From which location are prompts run, and can I set it per monitor?
- If API calls are used, is web search enabled?
- How often is each prompt run, and how are repeat runs aggregated?
- Are citations the retrieved pages the engine showed, or URLs parsed from generated text?
- How do you handle model or interface updates that shift results?
Question 5 is the one most vendors dislike. If a tool cannot explain it, treat the citation numbers as soft.
A practical setup for your own team
Pick the route per question
Use UI data for anything that goes into a board deck: share of voice, citation share, position against competitors, shopping and ads exposure. Use API data for cheap breadth and for the parametric-knowledge view.
Sample more than once
Track topics, not single prompts, and run each prompt repeatedly. Report ranges or averages across many responses instead of one screenshot. Jurenka's confidence-interval idea is the right end state, even if your tool only gives you averages.
Source prompts from real demand
A tracker is a sample, and the prompts you choose decide what the number means. Start from Search Console queries, sales calls, and prompt volume data rather than guesses.
Cross-check against what you control
If your tracked visibility rises but crawler logs show AI bots never fetch your key pages, or AI-referred traffic is flat, something is off. Crawler logs and visitor analytics are the sanity check on any prompt dataset. Promptwatch's AI crawler logs are one way to do this, and its Academy covers how it collects data if you want the specifics.
Don't compare numbers across methods
If you switch tools or routes, expect a break in your series. A move from an API-based tracker to a UI-based one will probably change your share of voice for reasons unrelated to real performance. Re-baseline, and annotate the chart.
What to do this week
- Run one commercial prompt in the real interface and through the vendor's API (or ask your vendor to). Compare cited sources and brands named. The gap is usually obvious in minutes.
- Check the run frequency in your current tool. If it is once per prompt per week, treat the trend lines as anecdotes.
- Run a free AI visibility report or a 7-day trial of a UI-based tool against your existing numbers, and see which story matches your crawler logs and traffic.
- Confirm pricing on each vendor's own page before you commit, since plans and prompt limits change. Promptwatch's are at promptwatch.com/pricing.
For wider context on how AI answers get built and cited, see Promptwatch Data, which publishes regularly updated research from the interfaces themselves.
Bottom line
The disagreement between UI and API data is not a rounding error. Sources overlap by roughly a quarter, brand counts inflate, citations get invented, and whole surfaces like shopping and ads disappear. Use the real interface for anything that represents buyer reality, use APIs where they are the only door, and insist on repeated sampling either way. One number from one run is just one draw from a distribution.




