DejanBot, the DEJAN web crawler
DejanBot is the web crawler operated by DEJAN. It requests single pages to confirm that links to our clients' sites are still in place and still point where they should. It identifies itself in every request as DejanBot/1.0 (+https://dejan.ai/bot), reads and obeys robots.txt before every fetch, and requests one page at a time from any one host with a pause between requests.
User agent
Every request carries this exact User-Agent header:
DejanBot/1.0 (+https://dejan.ai/bot)
The token to use in robots.txt rules is DejanBot. We do not send a browser user-agent string, and we do not use a residential or rotating proxy service. A request that claims to be DejanBot without this header did not come from us.
What it fetches
DejanBot requests specific URLs that we already hold in a list: pages a client's link is supposed to appear on. It does not follow links out of those pages, does not discover new URLs, and does not crawl a site section by section. A typical visit is one page, once a week.
It also fetches /robots.txt from each host, and caches the result for one hour.
Crawl rate
One request per host at a time, never in parallel. Between two requests to the same host, DejanBot waits for the Crawl-delay that host sets in robots.txt, up to 30 seconds. Where no Crawl-delay is set, it waits 2 seconds. Each page is checked about once a week.
What it stores
For each page it reads, DejanBot records the HTTP status code, the address it finished on after redirects, whether the link it was looking for was present, that link's anchor text and rel attribute, and how many links the page contains. Page text is discarded after the check. Nothing is republished, indexed or used to train a model.
How to block it
To stop DejanBot from requesting anything on your site, add this to your robots.txt:
User-agent: DejanBot Disallow: /
To slow it down instead, set a delay:
User-agent: DejanBot Crawl-delay: 30
A disallowed page is never fetched. When robots.txt blocks a page, or a server refuses the request, our record for that link reads "could not check" rather than "link missing".
Contact
Questions, complaints or a request to stop crawling: dejan.ai/contact.