Page hits counts requests to real pages from the same IP addresses that fetched this file. A zero there means the agent took the index and read nothing from it. IP addresses are shared within a company’s crawl fleet, so treat the figure as the fleet’s activity rather than that one named agent’s. Cadence is the mean gap between consecutive fetches.
| Agent | Fetches | Share | IPs | Days | Page hits | Cadence | Last seen |
|---|---|---|---|---|---|---|---|
| Browser | 85 | 54.1% | 44 | 19 | 240 | 5.1h | 14 Aug 07:10 |
| Unrecognised | 40 | 25.5% | 21 | 18 | 1202 | 11.1h | 15 Aug 00:08 |
| curl | 13 | 8.3% | 2 | 3 | 1173 | 11.9h | 02 Aug 05:45 |
| AhrefsBot | 4 | 2.5% | 4 | 4 | 1137 | 66.3h | 13 Aug 13:37 |
| HeadlessChrome | 4 | 2.5% | 4 | 2 | 8 | 9.3h | 14 Aug 07:10 |
| Googlebot | 3 | 1.9% | 3 | 2 | 0 | 18.6h | 05 Aug 18:28 |
| Meta-ExternalAgent | 2 | 1.3% | 2 | 2 | 114 | 31.3h | 14 Aug 14:02 |
| MJ12bot | 1 | 0.6% | 1 | 1 | 17 | — | 05 Aug 04:28 |
| SemrushBot | 1 | 0.6% | 1 | 1 | 26 | — | 08 Aug 19:30 |
| Applebot | 1 | 0.6% | 1 | 1 | 31 | — | 10 Aug 09:52 |
| ClaudeBot | 1 | 0.6% | 1 | 1 | 149 | — | 02 Aug 13:16 |
| Bingbot | 1 | 0.6% | 1 | 1 | 195 | — | 04 Aug 11:16 |
| Amazonbot | 1 | 0.6% | 1 | 1 | 223 | — | 04 Aug 05:26 |
Raw strings that matched no known crawler pattern. A new AI crawler shows up here first; add its name to the pattern list to give it a proper row above.
| User agent | Fetches | IPs | Last seen |
|---|---|---|---|
| AgentRadar-Research/1.0 (BCG Henderson Institute; agentic-web-research) | 6 | 4 | 11 Aug 14:35 |
| DataDingoBot/1.0 (+https://datadingo.au/bot) | 5 | 1 | 05 Aug 18:13 |
| BotWitness/1.0 (+https://bot-witness.com/bot; contact: hello@bot-witness.com) | 5 | 1 | 14 Aug 07:40 |
| PipericBot/1.0 (+https://piperic.com/bot) | 4 | 1 | 14 Aug 06:29 |
| LLMS-Txt-Scanner/1.0 (Research) | 3 | 2 | 09 Aug 03:13 |
| TheWebReport/1.0; +https://theweb.report | 3 | 2 | 13 Aug 01:08 |
| UMAIBot/1.0 (+https://umai.entelijan.com) | 2 | 1 | 07 Aug 06:13 |
| portal4Agents-agent-surface-scanner/0.1 | 2 | 1 | 12 Aug 07:10 |
| Mozilla/5.0 (compatible; pixelsforbots-index/1.0; +https://pixelsforbots.com/ai-readiness) | 2 | 1 | 15 Aug 00:08 |
| urltxtcrawl/1.0 | 1 | 1 | 13 Aug 23:48 |
| AIWebSignals/3.3.0 (+https://github.com/derailable/ai-web-signals) | 1 | 1 | 30 Jul 10:21 |
| ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:cmu-clueweb-crawler@andrew.cmu.edu) | 1 | 1 | 06 Aug 10:02 |
| Mozilla/5.0 | 1 | 1 | 27 Jul 21:39 |
| NetRanksPolicyAudit/1.0 (+https://netranks.ai) | 1 | 1 | 04 Aug 16:20 |
| UmaiBot/1.0 (+https://umai.entelijan.com/bot) | 1 | 1 | 03 Aug 20:10 |
| secretscan/1.0 (+oneshot) | 1 | 1 | 13 Aug 06:01 |
| AIWebSignals/3.2.0 (+https://github.com/derailable/ai-web-signals) | 1 | 1 | 29 Jul 17:25 |