How to Track AI Agents Visiting Your Website: A Practical 2026 Setup Guide

GA4 can't see AI bots because they don't run JavaScript. Here's how to track AI agent visits with server logs, CDN data, and verified IPs, and what to do with the numbers.

Key takeaways

  • AI agents and crawlers request pages straight from your server and don't run JavaScript, so GA4 and similar script-based analytics never record them.
  • The reliable sources are server logs, CDN logs, or a lightweight middleware layer. User-agent matching is the starting point, but user agents can be spoofed, so verify by IP range where you can.
  • Separate bots into three groups before you act: training crawlers, search/retrieval crawlers, and user-triggered agents. They need different robots.txt decisions.
  • Compare your logs against a public benchmark. If a big provider is near zero on your site, you are probably blocked.
  • Two weeks of data shows patterns. Two months shows trends.

Why your analytics can't see AI agents

Open GA4 and you'll see sessions, page views, and referrals. What you won't see is the ChatGPT fetch that pulled three paragraphs from your pricing page at 2 a.m., or the crawler that hit your blog 400 times last week.

The reason is simple. GA4 works by running a JavaScript tag in a browser. Most AI crawlers request the raw HTML and leave. No script runs, no event fires, nothing is recorded. It isn't undercounted. It's absent.

That has a second consequence for content. Because most AI crawlers don't render JavaScript, anything that only appears after a script runs is invisible to them too. Put your important content in the first HTML response. Semrush makes the same point in its guide to preparing your site for AI agents, citing Vercel's research on AI crawlers.

A note on the source material I looked at while researching this: one popular tutorial describes a "GA4 setup" for agents. What it actually describes is a server-side logging endpoint, which is the right idea, just not GA4. Keep that distinction in mind when you see advice on this topic.

WebMCP guide on tracking AI agent visits, showing the user agent table for retrieval agents

Know what you're looking for: three kinds of AI bot

Lots of guides lump every AI user agent into one table. That's a mistake, because the robots.txt decision is different for each group.

CategoryWhat it doesExamplesTypical decision
Training crawlersCollect content for model trainingGPTBot, ClaudeBot, Bytespider, Meta-ExternalAgentMany sites restrict these
Search and retrieval crawlersBuild indexes AI search uses to cite sourcesOAI-SearchBot, PerplexityBot, Claude-SearchBotUsually allow, since this is how you get cited
User-triggered agentsFetch a page live because a person askedChatGPT-User, Perplexity-User, Claude-UserUsually allow, these are real customers in a sense

The Nivarix guide to agent readiness frames this well: allow search discovery, allow legitimate customer agents, restrict training crawlers if you want to. Agent readiness is about control, not just openness.

Two entries that show up in many tables deserve a caution. Google-Extended and Applebot-Extended are generally described as robots.txt control tokens that govern how content is used for AI features, rather than crawlers with their own separate hits in your logs. Check Google's and Apple's own documentation before building a filter around them. If you search your logs for them and find nothing, that's probably why.

BrightEdge's agent guide also lists DuckAssistBot, Google-NotebookLM, and Meta-ExternalAgent, and reports that Bytespider activity rose 138% between November and February 2026. The bot list changes quickly, so treat any static table as a snapshot. The AI bots and web crawlers directory at promptwatch.com/bots tracks 400+ of them with robots.txt guidance.

Step 1: get access to raw request data

You need one of these. Pick whichever you can reach this week.

SourceEffortGood forWeakness
Web server logs (nginx, Apache)LowSmall sites, a quick baselineManual parsing, no IP verification by default
CDN logs (Cloudflare, CloudFront, Fastly, Vercel)Low to mediumMost sites, since bots hit the CDN firstLog retention and export limits vary by plan
Server middlewareMediumCustom logging with response codes and timingYou maintain the code
A dedicated crawler analytics toolLowOngoing monitoring, verified botsPaid, and one more tool

If you sit behind a CDN, use its logs. Cached responses may never reach your origin server, so origin logs can understate bot activity.

Step 2: filter by user agent

Start simple. Pull a day of logs and search for known AI strings.

grep -iE "GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User|Bytespider|Meta-ExternalAgent|DuckAssistBot" access.log \
  | awk '{print $1, $7, $9}' | head

That gives you IP, path, and status code for each hit. Count requests per agent and per path:

grep -iE "GPTBot|ClaudeBot|PerplexityBot" access.log \
  | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

That second command is the quick win. It shows which pages bots read most, and by omission, which pages they never touch.

Step 3: verify, because user agents lie

Anyone can send a request claiming to be GPTBot. If you only match strings, your numbers will include scrapers wearing a costume, and you may block or allow the wrong things.

The fix is to check the source IP against the ranges the operator publishes. OpenAI, Anthropic, Google, and Perplexity each publish IP lists or verification methods. Cloudflare also flags verified bots at the edge. Promptwatch Data's AI crawler traffic report uses exactly this method: it counts only requests whose IP matches the operator's published ranges, so spoofed user agents are excluded.

For a small site, a practical compromise is to run the string match first, then spot-check the top IPs for each bot with a reverse lookup or the vendor's published list.

Step 4: log what matters

If you build your own middleware or endpoint, record these fields for each bot request:

  • timestamp (UTC)
  • agent name and category (training, search, user-triggered)
  • verified or unverified
  • path
  • response code
  • response time

The response code is the most useful column. A bot that gets a 403, 429, or 5xx isn't reading your page, and it won't cite what it can't read. Plenty of sites discover they've been accidentally blocking a major provider through a WAF rule or a bot-protection default.

Store it wherever you already work. A database table is ideal. A daily script that appends to a CSV or Google Sheet is fine to start. Track date, agent, pages, total requests, and error count. After two weeks you can answer basic questions: which agents come most, and are there important pages with zero agent traffic?

Step 5: benchmark against the market

Your own numbers mean little without context. Promptwatch Data tracked IP-verified crawler requests from OpenAI, Anthropic, Google, Perplexity, and Mistral from June 8 to September 6, 2026. OpenAI accounted for 79.8% of verified requests in the week of August 31 to September 6, down from 94.8% in the week of June 8 to 14. The mix is shifting, so a static assumption about who crawls the web goes stale quickly.

The report's own tip is the useful part. If a provider has a large share in the benchmark but is near zero in your logs, you are probably blocked or unreachable for that provider. Check robots.txt, CDN rules, and WAF settings for every provider, not just the one you care about most.

Also keep in mind that a provider's share can be training traffic, answer-time fetches, or both. Break the numbers out by individual bot, not only by company.

Two other reports are worth a look, with the caveat that I'm only summarizing their catalog descriptions here. Promptwatch Data reports that Meta-WebIndexer grew from about 2% to nearly 38% of tracked AI crawler requests between mid-July and August 9, 2026, and there is a separate report on Claude's citation crawler growth from December 2025 to April 2026. If you've never searched your logs for Meta's bots, do it.

Tool options

You can do all of this with grep and patience. Many teams eventually want something that runs on its own.

Promptwatch includes IP-verified AI crawler logs (it calls the feature Agent Analytics) covering bots like GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Meta-WebIndexer. It shows which pages they read, what errors they hit, and a crawl-to-citation path per page, which helps you tell an access problem from a content problem. Setup works through Cloudflare, Vercel, Fastly, CloudFront, Netlify, Akamai, a custom endpoint, or the WordPress plugin. See the AI crawler logs page for details, and confirm current plan limits at promptwatch.com/pricing, since crawler logs are not on the Essential plan.

Favicon of Promptwatch

Promptwatch

Track and optimize your brand's visibility in AI search engines
View more
Screenshot of Promptwatch website

DarkVisitors is a narrower option focused on tracking AI agents, bots, and LLM referrals visiting a site.

Favicon of DarkVisitors

DarkVisitors

Track AI agents, bots, and LLM referrals visiting your websi
View more
Screenshot of DarkVisitors website

Otterly.AI has also launched an Agent Analytics feature aimed at the same blind spot, with a help center guide for setup. I haven't compared its depth against the others.

Favicon of Otterly.AI

Otterly.AI

Affordable AI visibility monitoring
View more
Screenshot of Otterly.AI website

If you'd rather crawl your own site than watch bots crawl it, a technical crawler shows what a bot would find.

Favicon of Screaming Frog

Screaming Frog

Industry-leading website crawler for technical SEO audits
View more
Screenshot of Screaming Frog website

For a broader list of options, the GEO software directory at bestgeosoftware.com is a good place to browse.

ApproachVerified IPsShows citationsCostBest for
grep on raw logsNo (manual)NoFreeA first baseline
Custom middlewareOnly if you build itNoDev timeTeams with engineers
DarkVisitorsCheck vendor docsNoCheck vendorBot tracking and robots.txt control
PromptwatchYesYes, per pagePaid plansClosing the loop from crawl to citation to traffic

Step 6: connect crawls to outcomes

Crawl counts alone can mislead. A bot reading a page doesn't mean the page gets cited. Three questions give you the full picture:

  1. Is the bot reaching the page? (Logs, status codes.)
  2. Is the page getting cited in AI answers? (Prompt and citation tracking.)
  3. Does anyone click through? (Visitor analytics.)

On the third point, be realistic. Semrush advises treating AI referral traffic as a secondary signal, because AI answers mostly shape whether a buyer hears about you before they ever land on your site. That influence often doesn't show up as a session. Referral data is also getting harder: ChatGPT stopped sending a referrer on October 8, 2026, so some visits now only show up if the link carries a utm_source=chatgpt.com tag. Don't build your whole measurement plan on referrals.

The ratio between crawls and citations tells you what to fix. Lots of crawls and no citations points at content, structure, or authority. No crawls points at access. Many errors points at your stack.

Step 7: act on what you find

A tracking setup that nobody acts on is a hobby. Here's a short list of fixes that logs commonly reveal.

  • Unblock the bots you want. Check robots.txt and WAF rules for retrieval and fetcher bots, not only Googlebot.
  • Decide on training crawlers deliberately. Blocking GPTBot won't remove you from ChatGPT search if OAI-SearchBot and ChatGPT-User can still reach you, but make sure that's the outcome you intend.
  • Fix errors. Repeated 403s, 429s, and timeouts on key pages are the cheapest wins.
  • Move key content into the initial HTML. If a page depends on client-side rendering, bots may see an empty shell.
  • Add structured data such as Organization and Product markup so facts are easy to extract.
  • Make pages stand alone. An agent may read one case study with no context, so each important page needs enough information to make sense by itself.
  • Don't bother with markdown copies for AI search. Promptwatch Data found markdown files make up just 0.05% of AI search citations, so markdown mostly serves agents, not AI search.

The free Agentic readiness check and robots.txt generator for AI are quick ways to see where your site stands before you touch any configuration.

A two-month plan

Week 1: pull a week of logs, filter by user agent, count requests per bot and per path.

Week 2: add status codes. List every page returning errors to a bot. Fix the top five.

Weeks 3 and 4: add IP verification or move to a tool that does it. Re-run the benchmark comparison.

Month 2: tie crawl data to citation data and decide which pages to improve, which to unblock, and which to leave alone.

Start this week. The data gets more valuable the longer you collect it, and a single day of logs can't show you a trend.

Common mistakes

  • Trusting GA4 to show bot traffic. It won't.
  • Treating every AI user agent as one group and writing a blanket allow or block rule.
  • Counting spoofed requests as real ones.
  • Reading only origin logs when a CDN serves most of your traffic.
  • Celebrating crawl volume without checking whether the pages get cited.

If logging, verification, and the follow-up content work sound like more than your team has time for, 1001 SEO Media runs technical SEO and GEO engagements that cover exactly this kind of setup.

Share:

© 2026 AI Search Tools · Best AI search tools and platforms · RSS

AI Search Tools is an affiliate review site. When you click links to vendors or buy through links on our site, we may earn an affiliate commission at no extra cost to you.

AI Search Tools is a review website based on user reviews on Reddit and G2, and on publicly available information. We keep everything as up to date as possible, but pricing and features can change. Always confirm the details with the vendor before purchasing.