In a 30-day server-log census of dejan.ai's three machine files, the big-three AI crawlers fetched robots.txt 1,150 times combined, llms.txt 4 times, and the OKF content bundle 23 times.
Between 27 July and 15 August 2026, dejan.ai logged every fetch of its three machine-readable files: /robots.txt (the crawl rules), /llms.txt (the llmstxt.org site index for language models), and the OKF bundle at /okf/ (the full machine-readable text of the site). Across the window robots.txt was fetched 13,442 times, the OKF bundle 2,841 times, and llms.txt 157 times. Counting only the crawlers of Google, Anthropic and OpenAI: 1,150 fetches of robots.txt, 23 of the OKF bundle, and 4 of llms.txt, none of them from OpenAI.
| File | Anthropic | OpenAI | All agents | |
|---|---|---|---|---|
| robots.txt | 243 | 608 | 299 | 13,442 |
| OKF bundle | 11 | 1 | 11 | 2,841 |
| llms.txt | 3 | 1 | 0 | 157 |
Google is Googlebot plus GoogleOther and Google-CloudVertexBot. Anthropic is ClaudeBot, Claude-User and anthropic-ai. OpenAI is GPTBot, OAI-SearchBot and ChatGPT-User.
13,442 fetches from 3,216 distinct IP addresses and 42 distinct agents. Anthropic's 608 fetches split into ClaudeBot (445) and Claude-User (163). OpenAI's 299 were OAI-SearchBot (295), GPTBot (1) and ChatGPT-User (3). Google's 242 came from Googlebot. Full snapshot: dejan.ai/i/21.
157 fetches from 78 distinct IP addresses and 13 distinct agents. The breakdown by agent:
| Agent | Fetches | Share |
|---|---|---|
| Browser | 85 | 54.1% |
| Unrecognised | 40 | 25.5% |
| curl | 13 | 8.3% |
| AhrefsBot | 4 | 2.5% |
| HeadlessChrome | 4 | 2.5% |
| Googlebot | 3 | 1.9% |
| Meta-ExternalAgent | 2 | 1.3% |
| ClaudeBot, MJ12bot, SemrushBot, Applebot, Bingbot, Amazonbot | 1 each | 0.6% each |
The three Googlebot fetches were followed by no page request from the same IP addresses. Full snapshot: dejan.ai/i/19.
2,841 fetches from 426 distinct IP addresses and 22 distinct agents, spread across 453 distinct files. Browser (2,208, 77.7%) and Unrecognised (541, 19.0%) account for most of it. Among AI crawlers: Google 11, OpenAI 11 (ChatGPT-User 6, GPTBot 3, OAI-SearchBot 2), Anthropic 1. Full snapshot: dejan.ai/i/20.
Fetches are counted from dejan.ai's own request log over the 30-day window ending 15 August 2026; logging for these files began 27 July 2026, so the window holds 20 days of data. Agents are grouped by the crawler's published User-Agent token; "Browser" is a request carrying a normal browser User-Agent, "Unrecognised" is a User-Agent that matches no known crawler or browser. The counts are fetches of the files, not renders, citations, or answer appearances.
Dan Petrovic ·
Aug 15, 05:02
