Who fetches it
Page hits counts requests to real pages from the same IP addresses that fetched this file. A zero there means the agent read the rules and requested nothing. IP addresses are shared within a company’s crawl fleet, so treat the figure as the fleet’s activity rather than that one named agent’s. Cadence is the mean gap between consecutive fetches.
| Agent | Fetches | Share | IPs | Days | Page hits | Cadence | Last seen |
|---|---|---|---|---|---|---|---|
| Unrecognised | 2514 | 18.7% | 714 | 20 | 19787 | 0.2h | 15 Aug 04:15 |
| Browser | 2407 | 17.9% | 825 | 20 | 650236 | 0.2h | 15 Aug 04:05 |
| MJ12bot | 2041 | 15.2% | 260 | 20 | 26135 | 0.2h | 15 Aug 04:15 |
| SemrushBot | 1842 | 13.7% | 60 | 20 | 176931 | 0.2h | 15 Aug 04:03 |
| DotBot | 648 | 4.8% | 2 | 20 | 17531 | 0.7h | 15 Aug 04:02 |
| ClaudeBot | 443 | 3.3% | 84 | 20 | 3885 | 1.0h | 15 Aug 03:20 |
| Twitterbot | 338 | 2.5% | 9 | 20 | 515 | 1.3h | 15 Aug 03:56 |
| YandexBot | 296 | 2.2% | 222 | 20 | 211 | 1.5h | 15 Aug 03:56 |
| OAI-SearchBot | 295 | 2.2% | 52 | 20 | 90 | 1.5h | 15 Aug 03:46 |
| facebookexternalhit | 286 | 2.1% | 214 | 19 | 465 | 1.5h | 14 Aug 20:11 |
| Googlebot | 242 | 1.8% | 35 | 19 | 56483 | 1.8h | 14 Aug 19:13 |
| Amazonbot | 228 | 1.7% | 5 | 16 | 468 | 1.9h | 14 Aug 19:50 |
| YouBot | 210 | 1.6% | 100 | 19 | 309 | 2.2h | 15 Aug 02:27 |
| Screaming Frog | 198 | 1.5% | 91 | 20 | 251 | 2.3h | 15 Aug 01:51 |
| AhrefsBot | 176 | 1.3% | 165 | 20 | 18073 | 2.6h | 15 Aug 02:25 |
| Claude-User | 163 | 1.2% | 1 | 20 | 724 | 2.8h | 15 Aug 03:19 |
| aiohttp | 146 | 1.1% | 3 | 13 | 183 | 1.9h | 15 Aug 02:13 |
| Bytespider | 139 | 1.0% | 130 | 20 | 167 | 3.2h | 15 Aug 00:14 |
| PerplexityBot | 127 | 0.9% | 8 | 20 | 1506 | 3.6h | 15 Aug 01:55 |
| Bingbot | 113 | 0.8% | 16 | 19 | 100 | 4.0h | 14 Aug 22:35 |
| DuckAssistBot | 113 | 0.8% | 57 | 19 | 54 | 3.9h | 14 Aug 19:46 |
| LinkedInBot | 76 | 0.6% | 7 | 14 | 127 | 5.8h | 14 Aug 14:44 |
| (no user agent) | 71 | 0.5% | 11 | 20 | 829 | 6.4h | 15 Aug 03:58 |
| Applebot | 67 | 0.5% | 59 | 19 | 1737 | 6.8h | 14 Aug 21:51 |
| DataForSeoBot | 34 | 0.3% | 1 | 17 | 1157 | 12.3h | 15 Aug 00:44 |
| python-requests | 27 | 0.2% | 12 | 7 | 32 | 11.7h | 14 Aug 18:34 |
| Barkrowler | 25 | 0.2% | 11 | 6 | 4242 | 16.3h | 14 Aug 09:05 |
| CCBot | 22 | 0.2% | 15 | 9 | 1285 | 18.0h | 14 Aug 11:49 |
| Apache-HttpClientsilent | 20 | 0.1% | 2 | 19 | 0 | 22.7h | 14 Aug 14:13 |
| curl | 18 | 0.1% | 4 | 5 | 4937 | 22.2h | 11 Aug 22:32 |
| Go-http-client | 18 | 0.1% | 13 | 7 | 3 | 17.9h | 09 Aug 17:10 |
| python-httpx | 16 | 0.1% | 7 | 9 | 86 | 24.2h | 14 Aug 16:30 |
| Qwantbot | 16 | 0.1% | 7 | 16 | 267 | 26.5h | 13 Aug 17:44 |
| PetalBot | 15 | 0.1% | 6 | 15 | 18469 | 29.6h | 13 Aug 20:51 |
| Sogou | 10 | 0.1% | 10 | 10 | 641 | 45.7h | 14 Aug 11:11 |
| HeadlessChrome | 5 | 0.0% | 5 | 3 | 32 | 74.7h | 14 Aug 07:10 |
| node-fetch | 4 | 0.0% | 1 | 4 | 10 | 110.2h | 14 Aug 20:00 |
| ChatGPT-User | 3 | 0.0% | 1 | 1 | 3 | 0.0h | 01 Aug 21:30 |
| Baiduspider | 2 | 0.0% | 2 | 2 | 604 | 130.7h | 08 Aug 07:33 |
| Java | 1 | 0.0% | 1 | 1 | 2 | — | 30 Jul 08:42 |
| GPTBot | 1 | 0.0% | 1 | 1 | 506 | — | 31 Jul 14:54 |
| GoogleOther | 1 | 0.0% | 1 | 1 | 37019 | — | 28 Jul 23:37 |
Unrecognised user agents
Raw strings that matched no known crawler pattern. A new AI crawler shows up here first; add its name to the pattern list to give it a proper row above.
| User agent | Fetches | IPs | Last seen |
|---|---|---|---|
| mwmbl/0.2.0 (https://github.com/mwmbl/mwmbl/ contact daoud@mwmbl.org (Mwmbl Crawler Operator)) | 382 | 1 | 15 Aug 04:11 |
| Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html) | 202 | 7 | 14 Aug 20:18 |
| Mozilla/5.0 (compatible; AffsignalCrawler/1.0; +https://affsignal.com/bot) | 185 | 4 | 15 Aug 02:11 |
| Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler) | 164 | 3 | 12 Aug 06:52 |
| Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler) | 138 | 1 | 14 Aug 23:49 |
| Mozilla/5.0 | 126 | 124 | 14 Aug 18:53 |
| Mozilla/5.0 (compatible; IbouBot/1.0; +bot@ibou.io; +https://ibou.io/iboubot.html) | 100 | 80 | 15 Aug 02:30 |
| ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:cmu-clueweb-crawler@andrew.cmu.edu) | 99 | 12 | 14 Aug 12:07 |
| Mozilla/5.0 (compatible; WebCrawler/1.0) | 76 | 3 | 14 Aug 20:12 |
| (none sent) | 71 | 11 | 15 Aug 03:58 |
| Mozilla/5.0 (compatible; Optimizer) | 59 | 41 | 14 Aug 10:33 |
| Mozilla/5.0 (compatible; LyonlBot/1.0; +https://lyonl.com/bot.html) | 56 | 2 | 13 Aug 01:56 |
| SearchEngineBot/0.1 (+https://example.com/bot) | 54 | 2 | 13 Aug 16:44 |
| LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so) | 48 | 1 | 14 Aug 04:34 |
| amazon-Quick-on-behalf-of-1f69047b | 38 | 3 | 15 Aug 03:45 |
| Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine) | 36 | 25 | 15 Aug 02:02 |
| Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/) | 35 | 12 | 15 Aug 01:37 |
| Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine) | 35 | 31 | 15 Aug 04:15 |
| AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index) | 34 | 7 | 07 Aug 10:48 |
| Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html) | 34 | 1 | 12 Aug 11:51 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0; | 31 | 30 | 15 Aug 00:49 |
| ClearSignalRadar/0.1 (+https://getclearsignal.io; public-source change monitor) | 22 | 21 | 15 Aug 03:55 |
| Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/) | 22 | 22 | 14 Aug 13:36 |
| tracker (+contact: alex@glasswing.cv) | 20 | 15 | 15 Aug 00:20 |
| Mozilla/5.0 (compatible; HubSpot Crawler; HubSpot Domain check; +https://www.hubspot.com) | 20 | 18 | 14 Aug 16:41 |
| amazon-Quick-on-behalf-of-1a8840c7 | 19 | 3 | 14 Aug 21:49 |
| Mozilla/5.0 (compatible; Dislike404Bot/1.0; +http://dislike404.com/bot/) | 19 | 1 | 14 Aug 07:50 |
| Mozilla/5.0 (compatible; dejan.ai AI Agent Access Checker; +https://dejan.ai/tools/agents/) | 17 | 1 | 04 Aug 09:41 |
| SofyaBot/1.0 (+https://sofya.co/bot) | 15 | 1 | 15 Aug 00:34 |
| ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers) | 14 | 2 | 06 Aug 02:10 |
| AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com) | 13 | 2 | 11 Aug 03:26 |
| KauaiBot runid=dev-test | 13 | 4 | 06 Aug 19:04 |
| Findisebot/1.0 (+https://findise.net) | 12 | 2 | 10 Aug 11:29 |
| MetaCrawler/1.0 (+https://metacrawler.com/bot) | 11 | 2 | 01 Aug 11:03 |
| Mozilla/5.0 (compatible; wpbot/1.4; +https://forms.gle/ajBaxygz9jSR8p8G9) | 10 | 10 | 15 Aug 02:10 |
| DataDingoBot/1.0 (+https://datadingo.au/bot) | 10 | 1 | 05 Aug 18:13 |
| txtbot (+https://txtbot.net) | 10 | 9 | 10 Aug 23:24 |
| PubLogic crawler - helping publishers get compensated by AI companies - https://publogic.ai/bot | 10 | 10 | 14 Aug 13:40 |
| ExaSearchBot | 9 | 9 | 02 Aug 22:09 |
| YisouSpider | 9 | 9 | 13 Aug 08:54 |
| Thinkers/0.9 | 9 | 1 | 13 Aug 11:12 |
| Polon/1.0 (+robots) | 8 | 1 | 14 Aug 09:17 |
| wozup-alpha/0.1a | 7 | 1 | 10 Aug 16:09 |
| Mozilla/5.0 (compatible; Reflectionbot/1.0; +https://reflection.ai/bot) | 6 | 2 | 14 Aug 15:07 |
| AgentRadar-Research/1.0 (BCG Henderson Institute; agentic-web-research) | 6 | 4 | 11 Aug 14:35 |
| Mozilla/5.0 (compatible; getdomaindata/1.0; +https://getdomaindata.com/bot) | 6 | 4 | 13 Aug 04:16 |
| ZoominfoBot (zoominfobot at zoominfo dot com) | 6 | 1 | 11 Aug 06:24 |
| fyndbot (robots; https://fynd.bot) | 6 | 3 | 13 Aug 10:30 |
| Mozilla/5.0 (compatible; FossickBot/1.0; +https://fossick.bot) | 6 | 1 | 05 Aug 04:49 |
| Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/bot | 6 | 6 | 12 Aug 04:51 |