Enter a domain and its robots.txt is read once and compared against 144 known crawlers, answer engines and agents. Every one gets a plain statement of who runs it and what it does with what it takes, alongside what your file currently says to it. Flip any of them and a rewritten robots.txt appears below, ready to paste.
