Key takeaways
- Ahrefs Bot Analytics (free during beta, on all paid plans) tracks 30+ bots including AI crawlers, but automatic setup requires a Cloudflare, Vercel, CloudFront, or Fastly integration, and the smoothest path needs Cloudflare Enterprise.
- Semrush's equivalent, Bot Analytics and Agent Analytics, launched in February 2026, is Enterprise-tier only. The $139-$299/mo SEO + AI Search plans most buyers sign up for do not include it.
- Neither tool distinguishes training crawlers (GPTBot, ClaudeBot) from retrieval/citation crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot), even though blocking one and not the other has very different consequences for AI visibility.
- A clean Semrush or Ahrefs site audit checks robots.txt syntax, not whether a WAF is silently turning away GPTBot or ClaudeBot while letting the SEO tool's own (allowlisted) crawler through.
- The crawler mix moves fast: Promptwatch data shows OpenAI's share of verified AI crawler requests fell from 94.8% to 79.8% in under three months, and Meta's web indexer went from roughly 2% to nearly 38% share in under a month. A one-off bot audit goes stale quickly.
Why this question even matters now
For most of the last decade, Ahrefs vs Semrush was a fight over two things: whose backlink index was bigger and whose keyword database was fresher. That's still a real fight. But in 2026 both companies are trying to convince you they've also solved an entirely different problem: showing you which AI bots are crawling your site, and whether they can actually reach the pages that matter.
That's a bigger claim than it sounds. "AI visibility tracking" (prompt monitoring, citation share, sentiment) is a different animal from "AI crawler tracking" (server logs showing GPTBot, ClaudeBot, and PerplexityBot hitting your pages, or getting blocked before they do). A lot of marketing copy blurs the two. This guide doesn't.
What Ahrefs actually shows you
Ahrefs introduced Bot Analytics as a standalone tool in its February/March 2026 feature update, pitched plainly as something that "shows every bot crawling your site and the pages they hit, so you can protect your crawl budget." It's still in beta and free on all paid plans, which is generous, but the catch is in the setup.
Bot Analytics sorts crawlers into 12 categories (Search Engine, AI Crawler, AI Assistant, AI Search, Security, and others), with six report views: Overview, Bots, Categories, Pages, Status Codes, and Site Structure. There's a one-click "AI bots" filter, and a Page Inspect feature that shows what a page looked like at the moment a bot crawled it. On paper that's a solid feature set.
The problem is getting the data in. Setup is server-side, not JavaScript-based, which is the right call technically, but it means you need a CDN integration: Cloudflare Logpush (requires Cloudflare Enterprise), a Cloudflare Worker (works on the free tier, but you deploy the script yourself), Vercel log drains (Pro or Enterprise only), or a CloudFront + Firehose + S3 setup. If you're not already paying for one of those enterprise-tier CDN products, the "free" bot tracking in Ahrefs comes with an unadvertised bill attached.
What Semrush actually shows you
Semrush took a different path. Its Bot Analytics (part of Site Intelligence) and Agent Analytics (part of its AI Optimization suite) launched February 2, 2026, and both are Enterprise-only. Bot Analytics covers 30 bots (20 search engine, 10 AI) with two report types: an overview of crawl patterns and a URL-level view of which pages got bot visits and how they responded. Agent Analytics narrows the lens to AI bots specifically, checking whether agents can reach key pages and how they parse the content once they're there.
Semrush also shipped Crawler Profiles in late March 2026, which lets you simulate how a specific AI agent, starting with ChatGPT's crawlers, would interpret your site. Worth noting: that's a simulation, not a real log-based measurement. It's useful for testing changes before you ship them, but it won't tell you what actually happened last Tuesday.
Here's the part that trips people up. Semrush's SEO + AI Search tiers (SEO at $139/mo, Starter at $199/mo, Pro+ at $299/mo) include AI visibility tracking: prompt monitoring, AI sentiment, AI-ready Site Audit. None of that is log-based bot tracking. If you want Bot Analytics or Agent Analytics, you need the Enterprise tier, which third-party pricing trackers put around $18,000+ a year. That's a meaningful gap between what the marketing implies at $199/mo and what you actually get.
Side by side
| Capability | Ahrefs Bot Analytics | Semrush Bot/Agent Analytics |
|---|---|---|
| Availability | Free, beta, on all paid plans | Enterprise tier only (~$18K+/yr) |
| Bot coverage | 12 categories, AI filter included | 30 bots (20 search, 10 AI) |
| Setup requirement | Cloudflare / Vercel Pro / CloudFront / Fastly; full automation needs Cloudflare Enterprise | Log upload or Enterprise Site Intelligence integration |
| Crawler simulation | Not offered | Crawler Profiles (ChatGPT, more coming) |
| Training vs retrieval crawler split | No | No |
| Links bot data to content actions | No closed loop to content recommendations | Agent Analytics ties access issues to AIO workflow, limited |
| Data retention | Varies by plan | 3 months fine-grained, up to 10GB/day ingest |
The thing both platforms quietly skip
This is the part that matters most if you're actually trying to manage AI visibility, not just check a box. Neither Ahrefs nor Semrush, in their standard audits, distinguishes between training crawlers and retrieval crawlers. GPTBot trains OpenAI's models. OAI-SearchBot is what actually fetches pages to answer a live ChatGPT search query. Blocking the first opts you out of training. Blocking the second removes you from citations in ChatGPT's search results right now. Those are not the same decision, but most robots.txt files, and most SEO tool audits, treat them as one.
One stat worth sitting with: as of mid-2025, only about 14% of the top 10,000 domains had any AI-specific robots.txt rules at all. Most sites are making this call by accident, not by choice.
There's a second blind spot. A clean Site Audit from either platform tells you your robots.txt is syntactically valid. It does not tell you whether a WAF rule is quietly turning GPTBot or ClaudeBot away at the edge while letting Ahrefs' or Semrush's own crawler through, because that crawler is usually allowlisted. I've seen this exact scenario play out: a site passes every technical SEO check, and is still functionally invisible to the AI crawlers that matter, because nobody checked per-bot access outcomes separately from robots.txt validity.
Why a one-time audit isn't enough
Even if you get the setup right, the crawler landscape doesn't sit still long enough for a quarterly check-in to be useful. Promptwatch data from its production crawler logs shows OpenAI's share of verified AI crawler requests dropped from 94.8% in early June 2026 to 79.8% by early September, a meaningful diversification in under three months. Meanwhile Meta's web indexer went from roughly 2% of tracked crawler requests in mid-July to nearly 38% by August 9, becoming the single heaviest AI crawler in the dataset almost overnight, reportedly tied to Meta building its own web index so Meta AI doesn't depend on Google (promptwatch.com/data/meta-web-indexer). Claude's citation crawler told a similar story on a longer timeline: from about 30 visits a day in mid-December 2025 to several thousand a day by mid-April 2026 (promptwatch.com/data/claude-citation-crawler-visits-over-time).
The practical implication: a robots.txt rule or WAF config that was harmless six months ago can silently cut you off from a crawler that's now a major citation source. If your bot-tracking setup only refreshes when someone remembers to log in, you'll find out about the block months after it started costing you visibility.
What dedicated AI crawler tools add
A few tools exist specifically to answer the narrower question: can AI crawlers actually reach my site, bot by bot, right now. AI Crawler Check is a good example. It checks 248 individually-named AI and web crawlers across 8 categories, validates llms.txt (not just robots.txt), flags WAF blocking explicitly instead of leaving it as a silent gap, and distinguishes training crawlers from retrieval crawlers as a core part of its scoring. It doesn't do keyword research or backlink analysis, and it isn't trying to; it's a 42-check citation-readiness score, 0 to 100, on up to 20 URLs per batch.
That's a fair trade for most teams: keep Ahrefs or Semrush for what they're genuinely good at (keyword research, backlink index, rank tracking, full-site crawls) and add a narrower tool for the AI crawler layer specifically.
Purpose-built AEO platforms like Profound make a similar case from a different angle, arguing that infrastructure-level AI crawler tracking tied directly to content recommendations and attribution is a different product category from an SEO suite with an AI tab bolted on.
Where this leaves you
If AI crawler tracking is a nice-to-have feature inside a tool you already pay for keywords and backlinks with, Ahrefs Bot Analytics is the more accessible option right now, free during beta, though you'll want a Cloudflare or Vercel setup already in place to get the automated version working. Semrush's version exists but sits behind an Enterprise price tag that puts it out of reach for most mid-market teams, even ones already paying $299/mo for Pro+.
If AI crawler visibility is the actual priority, not a feature you stumble into, a dedicated tool will go deeper: per-bot access verification, llms.txt validation, training-vs-retrieval distinctions, and WAF detection that general-purpose site audits weren't built to catch. For teams going further and tracking how that crawler activity translates into actual citations, traffic, and revenue from AI platforms, a GEO-focused platform like Promptwatch layers crawler logs (Agent Analytics), citation data, and content generation on top of each other, closing the loop that a bot-tracking add-on alone can't.
For a wider look at tools built specifically around this problem, the GEO software directory at bestgeosoftware.com is a reasonable starting point, and ai-rank-tools.com covers the rank-tracking side of the same shift.
Either way, don't mistake a passing Semrush or Ahrefs site audit for proof that AI crawlers can reach your content. Those checks were built for a different question, and the crawler landscape is moving faster than either platform's standard audit cycle.