Build a robots.txt file from allow and disallow rules — set user-agent, add path rules, include a sitemap URL, and copy the generated file. Updates live as you edit.
Frequently Asked Questions
What is a robots.txt file?
robots.txt tells search engine crawlers which URLs they may or may not request from your site. It lives at the root of your domain (e.g. example.com/robots.txt).
What is the difference between Allow and Disallow?
Disallow blocks crawlers from accessing a path. Allow explicitly permits access — useful for overriding a broader Disallow rule on a subdirectory.
Should I include a sitemap URL?
Yes. Adding a Sitemap directive helps search engines discover your sitemap.xml. Validate your sitemap with the Sitemap Validator.
Does robots.txt guarantee pages won’t be indexed?
No. robots.txt controls crawling, not indexing. Pages blocked from crawling may still appear in results if linked from elsewhere.
Is this tool free and does it work on mobile?
Yes on both counts. The Robots.txt Generator on draft21 is completely free with no sign-up required and works on all devices.