Watch: Google, Anthropic and OpenAI crawlers fetched dejan.ai's robots.txt 1,150 times and its llms.txt 4 times in 30 days
In a 30-day server-log census of dejan.ai's three machine files, the big-three AI crawlers fetched robots.txt 1,150 times combined, llms.txt 4 times, and the OKF content bundle 23 times.
Transcript
How often do major AI crawlers actually look at the files meant to guide them? Over a twenty-day period, one website tracked every single request for three of its machine-readable files: the standard robots dot txt crawl rules, a language model index called llms dot txt, and a full-text bundle of the site.
The classic robots dot txt file was by far the most popular, with over thirteen thousand fetches. The full-text bundle was requested nearly three thousand times, while the newer llms dot txt file received just over one hundred and fifty hits.
When looking specifically at the big three AI giants—Google, Anthropic, and OpenAI—the numbers tell an interesting story. These crawlers checked the robots dot txt rules over eleven hundred times. But they barely touched the files specifically designed for AI. They requested the full-text bundle just twenty-three times, and the llms dot txt file only four times. In fact, OpenAI did not fetch the llms dot txt file even once.
Instead of advanced AI bots, the vast majority of traffic to the newer machine-readable files came from regular web browsers and unrecognized sources. It seems that while the tools for AI-friendly web indexing exist, the major players are still mostly relying on traditional crawling methods.