Watch: AI Crawlers
The bots AI systems use — training-data scrapers versus agentic assistants — that site owners must decide whether to allow or block.
Transcript
Every day, automated AI crawlers scour the web, but before you decide to block them, it helps to understand who they are. Not all bots are created equal, and blocking too broadly can accidentally shut out the good traffic that actually sends visitors to your site.
AI bots generally fall into two categories. First, there are training-data scrapers. These bots, like OpenAI’s GPTBot or Anthropic’s ClaudeBot, harvest text and images at a massive scale to train their models. You can easily identify most of them and block them using your site's robots.txt file if you want to protect your content.
The second group consists of agentic bots. Instead of passively gathering data, these are autonomous systems designed to perform multi-step tasks, mimicking how a human actually uses the web.
This leaves website owners with a tough strategic choice. If you block the scrapers to protect your intellectual property, you risk becoming completely invisible to the next generation of AI search engines and assistants. Finding the right balance is the key to staying visible in an AI-driven world.