The bots AI systems use — training-data scrapers versus agentic assistants — that site owners must decide whether to allow or block.
AI crawlers are the automated bots that AI systems use to access the web, and telling them apart matters before you decide whether to block them. Block too broadly and you can shut out the "good" traffic — search and assistants that send real visitors — while trying to keep out bulk scrapers.
There are two broad kinds. Training-data scrapers crawl at scale to collect text and images for model training — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot (Common Crawl), PerplexityBot and many more, most identifiable by user agent and blockable via robots.txt. Agentic bots are autonomous systems that plan and act on multi-step tasks, mimicking human workflows rather than passively harvesting data.
The strategic choice is real: blocking scrapers can protect content but also risks invisibility to the next generation of AI answers — the tension our CAPS proposal tries to resolve. It's closely tied to how AI agents and grounding systems consume the web.
