AI Bot Detector - Identify AI Crawlers by User-Agent

What is an AI crawler?

An AI crawler is an automated bot that fetches public web pages to feed a large language model or an AI search product. Unlike a classic search-engine crawler that exists to rank pages in results you click through to, AI crawlers often consume your content to train models or to answer a question directly - so the visit may never send a human to your site.

AI crawlers vs. search-engine crawlers: what's the difference?

Search crawlers (Googlebot, Bingbot) index pages so people find and visit them; AI crawlers fetch pages to train models or generate answers, often without a referral click. The practical split is by purpose:

• Training crawlers - collect text to train future models (GPTBot, ClaudeBot, CCBot, Amazonbot, Meta-ExternalAgent).
• AI search crawlers - index pages to answer queries inside an assistant (OAI-SearchBot, PerplexityBot, Applebot).
• User-initiated fetchers - load one page because a human asked the assistant about it (ChatGPT-User, Claude-User, Perplexity-User).
• Training-control tokens - not bots at all, but robots.txt labels (Google-Extended, Applebot-Extended) that opt you out of training while normal indexing continues.

How do I block or allow AI crawlers in robots.txt?

List each crawler's User-Agent token in robots.txt with a Disallow rule to block it, or a permissive rule to allow it. Robots.txt is voluntary - well-behaved bots obey it, but it is a request, not enforcement.

1. Block OpenAI training: User-agent: GPTBot then Disallow: /.
2. Block Anthropic training: User-agent: ClaudeBot then Disallow: /.
3. Opt out of Google's Gemini training but keep Search: User-agent: Google-Extended then Disallow: /.
4. Block Common Crawl (used by many models): User-agent: CCBot then Disallow: /.
5. For bots that ignore robots.txt (e.g. Bytespider has been widely reported to), block by IP/ASN at your CDN or WAF.

Can a User-Agent be faked?

Yes. A User-Agent is just a text header the client chooses, so any scraper can send GPTBot/1.0 while having nothing to do with OpenAI. Reliable detection therefore needs more than a string match:

• Reverse DNS (rDNS) - resolve the connecting IP to a hostname and confirm it ends in the vendor's domain, then forward-resolve that hostname back to the same IP (forward-confirmed rDNS).
• Published IP ranges - major vendors publish the IP/ASN ranges their crawlers use; verify the request originates there.
• Rate and behavior - genuine crawlers honor crawl-delay and back off; spoofers often do not.

This tool does signature matching only, so treat a match as "claims to be," not "proven to be."