Robots.txt Generator
Create a correct robots.txt in a minute.
Ignored by Google; used by Bing and Yandex.
● Everything runs in your browser. Your content and URLs are never sent to our servers.
How it works
robots.txt sits at the root of your site (example.com/robots.txt) and tells crawlers which URLs not to fetch. It's not a security measure — pages listed in it can still be indexed if linked from elsewhere (use a noindex tag for that) and anyone can read the file. Paths are case-sensitive and match by prefix; * matches any characters and $ the end of the URL. AI companies' crawlers respect their own user-agent names, which you can block individually without affecting search engines.