AI Crawlers
The bots AI systems use — training-data scrapers versus agentic assistants — that site owners must decide whether to allow or block.
Every day, automated AI crawlers scour the web, but before you decide to block them, it helps to understand who they are. Not all bots are created equal, and blocking too broadly can accidentally shut out the good traffic that actually sends visitors to your site.
AI bots generally fall into two categories. First, there are training-data scrapers. These bots, like OpenAI’s GPTBot or Anthropic’s ClaudeBot, harvest text and images at a massive scale to train their models. You can easily identify most of them and block them using your site's robots.txt file if you want to protect your content.
The second group consists of agentic bots. Instead of passively gathering data, these are autonomous systems designed to perform multi-step tasks, mimicking how a human actually uses the web.
This leaves website owners with a tough strategic choice. If you block the scrapers to protect your intellectual property, you risk becoming completely invisible to the next generation of AI search engines and assistants. Finding the right balance is the key to staying visible in an AI-driven world.
AI crawlers are the automated bots that AI systems use to access the web, and telling them apart matters before you decide whether to block them. Block too broadly and you can shut out the "good" traffic — search and assistants that send real visitors — while trying to keep out bulk scrapers.
There are two broad kinds. Training-data scrapers crawl at scale to collect text and images for model training — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), PerplexityBot and many more, most identifiable by user agent and blockable via robots.txt. Agentic bots are autonomous systems that plan and act on multi-step tasks, mimicking human workflows rather than passively harvesting data.
The strategic choice is real: blocking scrapers can protect content but also risks invisibility to the next generation of AI answers — the tension our CAPS proposal tries to resolve. It's closely tied to how AI agents and grounding systems consume the web.