robots.txt generator
Pick a template, block pages you don't need and AI bots, add your sitemap. Copy or download the finished file.
1Template
2All robots
User-agent: *One path per line. An asterisk * matches any characters, $ marks the end of the URL.
A pause between robot requests. Bing and some other robots follow it, Google and Yandex ignore this line.
3AI bots
4Rules for specific bots
For example, block SEO analytics crawlers or keep your images out of Google Images.
5Sitemap
robots.txt
Put the file in the root of your site so it opens at an address like example.com/robots.txt.
Robots.txt in a couple of minutes
- 1Pick a templatefor your platform or a regular website
- 2Set up the rulesblock unneeded sections, AI bots and specific robots
- 3Upload the filedownload robots.txt and put it in the root of your site
What robots.txt is
It's a text file in the root of your site that robots read first. It says which sections may be crawled and which may not, and where the sitemap is. Each block starts with a User-agent line, the robot's name, and an asterisk means all robots. Below it come the Disallow (block) and Allow rules.
Keep in mind: robots.txt controls crawling, not indexing. A page blocked in robots.txt can still show up in search if other sites link to it. To keep a page out of search for sure, leave it open to the robot and add a noindex meta tag.
How to block AI bots
AI companies crawl websites with their own robots: GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl, Bytespider from ByteDance. Some collect texts to train models, others look up an answer to a user's question right now. If you block training only, ChatGPT, Claude and Perplexity can still show links to your site.
Google-Extended and Applebot-Extended are not separate robots but tokens: they stop Google and Apple from using your texts for their AI, without affecting regular search. Remember that robots.txt is a request, not a lock: well-behaved bots follow it, but it can't protect private data.
Common mistakes
The most dangerous one is a Disallow: / line left in the block for all robots after development: the site drops out of search. Don't block CSS and JavaScript files, without them Google can't see the page the way a person does. Remember that a robot with its own block of rules no longer follows the general rules for all robots. After uploading, check the file in Google Search Console and Yandex Webmaster.
FAQ
Didn't find an answer? Message us and we'll help.
In the root of your site, so it opens at an address like example.com/robots.txt. Robots won't find a file in a subfolder. Each subdomain needs its own robots.txt. WordPress, Shopify and Tilda create the file automatically, you can replace it with your own or edit it in the settings.
Not always. It stops the robot from visiting the page, but if other pages link to it, the address can still appear in results without a description. To remove a page from search, use the robots meta tag with noindex and don't block it in robots.txt, or the robot will never see that tag.
Turn on "Block AI bots" and choose "Model training only" or "All AI bots". The file will list GPTBot, ClaudeBot, Google-Extended, CCBot and other known bots. Big companies respect this rule, but robots.txt gives no protection against those who ignore it.
No. Regular Google search works through Googlebot, while Google-Extended only controls whether your texts are used in Google's AI. But if you block AI search (OAI-SearchBot, PerplexityBot, Claude-SearchBot), your site will stop appearing in those services' answers.
Disallow blocks crawling of a path and everything that starts with it, Allow makes an exception inside a blocked section. When rules conflict, Google picks the longer, more specific one. For example, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php block the admin area but keep one needed file open.
It's a good idea. The Sitemap line tells every search engine where your sitemap is, even those where you never added it by hand. The address must be full, with https:// and the domain. You can list several sitemaps, each on its own line.
Try also
Also free, no sign-up