DejanBot
DejanBot is a friendly and well-behaved web crawler which visits a page and looks for broken links.
User agent
Every request carries this exact User-Agent header:
The token to use in robots.txt rules is DejanBot. We do not send a browser user-agent string, and we do not use a residential or rotating proxy service.
IP address
A user-agent string can be copied by anyone, so check the address the request came from. Every DejanBot request comes from one fixed IP:
A request from any other address is not DejanBot, whatever user-agent it sends. If this address ever changes, this page changes with it.
Crawl rate
One request per host at a time, never in parallel. Between two requests to the same host, DejanBot waits for the Crawl-delay that host sets in robots.txt, up to 30 seconds. Where no Crawl-delay is set, it waits 2 seconds. Each page is checked about once a week. It reads HTML only, and stops reading a page after 3 MB. A typical visit is one page, once a week. It also fetches /robots.txt from each host, and caches the result for one hour.
What it records
For each page it reads, DejanBot records the HTTP status code, the address it finished on after redirects, whether the link was found, that link's anchor text and rel attribute, and how many links the page holds. That is the whole record: a handful of fields about one link.
What it does not do
DejanBot does not keep page content. The text of the page is read once, in memory, to find the link, and is discarded when the check finishes. Nothing it reads is stored as a copy, republished, indexed, resold, or used to train a model. It runs no JavaScript, submits no forms, signs in to nothing, and carries no cookies from one page to the next. It requests pages that any visitor can request.
How to block it
To stop DejanBot from requesting anything on your site, add this to your robots.txt:
To slow it down instead, set a delay:
A disallowed page is never fetched. When robots.txt blocks a page, or a server refuses the request, our record for that link reads "could not check" rather than "link broken".
Contact
Questions, complaints, or a request to stop crawling: dejan.ai/contact. If DejanBot is causing trouble on your site, tell us and we will stop.