wikis / Agentic SEO / wiki / summaries / bot-and-agent-traffic-reports.md view as markdown report a mistake
Bot and Agent Traffic Reports
Key Points
- Every headline number has a different denominator. "AI bot traffic", "crawler requests", "all traffic", "AI page fetchers on European sites", and "AI scrapes" are separate populations. A claim that "X% of bot traffic is training" is only comparable to another figure if both count the same bots over the same network and period.
- Most reporters sell bot management or monetisation. Cloudflare, Fastly, and Akamai sell bot control; TollBit sells bot paywalls and licensing; HasData sells scraping APIs; TechnologyChecker sells a technographics database. Wiki analysis: all of them have a relevant commercial interest.
- The training share depends on how bots are classified. Cloudflare's own series moved between reports (see conflict below).
- robots.txt non-compliance is measured differently by every source, and each treats user-triggered fetchers differently. See user triggered fetchers and robots.
- Referral returns are tiny relative to crawling, but the ratio depends on counting only web referrals (native apps send no
Refererheader, which Cloudflare says may overstate its ratios).
Metric-scope table
| Publisher (vendor interest) | Date / period | Population counted | Headline number |
|---|---|---|---|
| Cloudflare Radar crawl/refer ratios | 19-26 June 2025 | HTML requests from a platform's crawler user agents divided by HTML requests whose Referer is that platform's hostnames; Cloudflare customer base |
Anthropic 70,900:1 to Mistral 0.1:1; Google's ratio fell 19.4% week over week |
| Cloudflare Radar purpose breakdown | first 28 days of July 2025 | AI-bot requests across Cloudflare, by declared purpose (Training, Search, User action, Undeclared) | Training "nearly 80%" of AI-bot crawling; Cloudflare says User action and Undeclared show daily patterns and (ambiguous "they") "account for less than 5% of AI bot traffic"; ChatGPT-User nearly three quarters of request traffic in the User action cohort (first 28 days of July 2025) |
| Cloudflare "Content Independence Day, one year on" | data to June 2026 | "Crawlers Cloudflare identifies by purpose" | 52% of crawler requests for AI training (June 2026) vs 22% (Spring 2025); mixed-use crawlers over 36%; over 50% of Internet traffic non-human; Google about 88% of referral traffic |
| TechnologyChecker.io re-analysis of Cloudflare Radar (Q3 2026 update, data through Sept 2026, Sept provisional) | July 2025 vs July 2026 | AI-bot requests by purpose, on Radar's revised series | Training 35.74% to 44.54%; Mixed Purpose 54.61% to 39.99%; Search 7.79% to 11.57%; User Action 1.38% to 2.66% |
| Fastly Q2 2025 Threat Insights (press release 19 Aug 2025) | mid-April to mid-July 2025 | AI bot traffic on Fastly; 6.5 trillion monthly requests, 130,000+ applications and APIs | Crawlers almost 80% and fetchers 20% of AI bot traffic; Meta 52% of AI crawler traffic, Google 23%, OpenAI 20%; OpenAI 98% of fetcher requests; peak fetcher load 39,000 requests per minute; automated bots 37% of observed activity |
| Vercel with MERJ | past month before publication; raw copy undated | Vercel network | GPTBot 569M fetches, Claude 370M, AppleBot 314M, PerplexityBot 24.4M, Googlebot 4.5 billion |
| TollBit State of the Bots Q3-Q4 2025 (as cited in TollBit blog, 8 Apr 2026, and Digiday) | H2 2025 | AI bot scrapes on TollBit's monitored sites | About 30% violated explicit robots.txt restrictions; ChatGPT-User 42% non-compliance; AI referral click-through fell from 0.8% to 0.27% over 2025 |
| TollBit State of the Bots H1 2026 (via SEJ and ppc.land, 14 Aug 2026; second-hand) | first half of 2026 | Identified AI page fetchers on European sites | About 15% reached URLs the site disallowed; any request to a disallowed URL counts as a bypass |
| Akamai "Protecting Publishing" (via Digiday, 13 Apr 2026) | calendar 2025; July-Dec 2025 for mix | AI bot activity on Akamai's network | 300% surge in 2025; commerce 48% and media 13% of targeting; in media, training crawlers 63% and fetchers 24% of AI bot types |
| HUMAN Security "2026 State of AI Traffic & Cyberthreat Benchmarks" (reported only inside the Digiday article) | calendar 2025 | Automated vs human traffic on HUMAN's platform | Automated traffic growing 8x faster than human; AI scraper traffic +597% January to December 2025; AI-driven traffic +187% in 2025 |
| Raptive (via Digiday) | date not given (a recent guide) | Raptive's network of 6,000 independent publishers | ChatGPT drives under 0.2% of traffic |
| HasData AI Crawler Block Index | July 2026 baseline, updated 26 Oct 2026 | 10,894 domains (9,746 Tranco top-web plus 1,148 news sites); enforcement probes on 2,096 | 39.5% of sites that disallow GPTBot served it a live 200 from a datacenter IP |
The HUMAN row is not from a HUMAN document; this wiki did not gather one. Treat it as Digiday's account.
Conflict: what share of AI crawling is "training"?
Three Cloudflare-derived figures appear in the sources and are not interchangeable:
- Cloudflare, July 2025 window: training "nearly 80%" of AI-bot crawling.
- Cloudflare, mid-2026: 52% of crawler requests for training as of June 2026, "up from 22% in Spring 2025", with over 36% mixed-use. A different denominator (crawlers identified by purpose) and a different direction from the 80% figure.
- TechnologyChecker's pull of Radar: training 35.74% in July 2025 and 44.54% in July 2026. It states Radar revised its crawl-purpose series ("it moved volume from Search into Training across its whole history before this pull"), so figures do not match earlier editions. In that pull, Training plus Mixed Purpose fell from 90.35% to 84.53% over the year.
Wiki analysis: the sources do not reconcile these. Use one report's own series for trends, and name the series when quoting a share.
Conflict: crawl-to-refer for one operator
Cloudflare Radar's Anthropic ratio was 70,900:1 for 19-26 June 2025. TechnologyChecker's Q1 2026 headline: ClaudeBot 20,583:1 and OpenAI 1,255:1. Its Q3 table lists Anthropic at 36,551:1 (Q3 2025), 5,143:1 (Q2 2026) and 988:1 (Q3 2026, provisional), with Perplexity at 1,083:1 in Q3 2026. Windows differ (a week, a quarter), so these show volatility more than a trend.
Things the reports say about enforcement
- Cloudflare's July 2026 policy (as summarised by ppc.land): from 15 September 2026, new domains onboarding to Cloudflare block Training and Agent crawlers by default on pages that display ads; Search stays allowed.
- TollBit notes Google-Agent is a user-triggered fetcher that "generally ignores robots.txt", while publishing IP ranges in
user-triggered-agents.jsonand a reverse DNS patterngoogle-proxy-***-***-***-***.google.com. - Digiday quotes Akamai that "AI fetchers" are growing and that third-party scrapers form a separate layer (a media analyst counted 21 vendors; TollBit counts nearly 40).
Relevant Concepts
- ai crawlers and user agents: who the bots are, by purpose class.
- user triggered fetchers and robots: why fetcher numbers are contested.
- robots txt for ai: what disallow lines can and cannot do.
- cloudflare: product context for the Radar figures.
- paid access x402 and pay per crawl: the monetisation response.
Source Metadata
| File | Publisher | Date shown | Type |
|---|---|---|---|
| raw/web_community-a-deeper-look-at-ai-crawlers-breaking-down-traffic-by-purpos.md | Cloudflare blog | data July 2025 | Vendor report |
| raw/web_community-the-crawl-before-the-fall-of-referrals-understanding-ai-s-im.md | Cloudflare blog | data June 2025 | Vendor report |
raw/web_community-content-independence-day-one-year-on-building-the-business-m.md (a second copy, -m-2.md, differs and was not used) |
Cloudflare blog | data to June 2026 | Vendor report; draws on Radar and Cloudflare's Investor Day 2026 presentation |
| raw/web_community-we-analyzed-robots-txt-across-cloudflare-s-network-publisher.md | TechnologyChecker.io | published 3 Apr 2026, updated 1 Oct 2026 | Third-party analysis of Radar data |
| raw/web_community-new-fastly-threat-research-reveals-ai-crawlers-make-up-almos.md and raw/web_community-ai-bots-in-q2-2025-trends-from-fastly-s-threat-insights-repo.md | Fastly | 19 Aug 2025 | Vendor report |
| raw/web_community-google-just-told-the-web-what-s-coming-agents-that-don-t-fol.md | TollBit blog | 8 Apr 2026 | Vendor commentary citing its own report |
| raw/web_community-15-of-ai-page-fetchers-in-europe-reached-disallowed-urls-tol.md | ppc.land (relaying SEJ and TollBit) | 14 Aug 2026 event | Secondary coverage |
| raw/web_community-publishers-face-growing-ai-bot-and-third-party-scraper-activ.md | Digiday | 13 Apr 2026 | Trade press relaying Akamai, HUMAN, Raptive, TollBit, MSG |
| raw/web_community-the-rise-of-the-ai-crawler-vercel.md | Vercel and MERJ | date not shown in raw copy | Vendor study |
| raw/web_community-the-ai-crawler-block-index-hasdata.md | HasData | updated 26 Oct 2026 | Vendor study |
Not gathered: the primary HUMAN, TollBit, and Akamai reports, and any DataDome report. Numbers attributed to them above are as relayed by the files listed.