Writing the file
Edit the first group
It starts as
User-agent: *with/admin/,/checkout,/basketand/searchdisallowed. Add or remove paths, one per line.Add exceptions with Allow
Put a more specific path in the Allow box to open part of a blocked folder, such as a help page inside
/account/.Add groups for particular crawlers
Add a crawler group creates a new block, starting as Googlebot; change the name to any user agent. Remove the last group takes one away.
Decide on AI training crawlers
Tick Block AI training crawlers to add whole-site blocks for GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot.
List your sitemaps and download
One address per line. They are written at the end of the file. Download
robots.txtand upload it to the root of the site.
Example: a craft supplies shop with a private uploads folder
An online craft shop (fictional) keeps customer account pages out, except for its account help page, and does not want Google Images to pick up files in a private uploads folder. Two groups and one sitemap produce the file below. Allow lines are always written before Disallow lines within a group; order does not change the result, because the longest matching rule wins.
Group 1: User agent * Disallow: /account/, /basket, /search Allow: /account/help Group 2: User agent Googlebot-Image Disallow: /uploads/private/ Sitemap: https://www.pennycrafts.example/sitemap.xml
User-agent: * Allow: /account/help Disallow: /account/ Disallow: /basket Disallow: /search User-agent: Googlebot-Image Disallow: /uploads/private/ Sitemap: https://www.pennycrafts.example/sitemap.xml
Before uploading
- No
Disallow: /underUser-agent: *unless you really mean to hide the whole site from search. - CSS, JavaScript and image folders are not blocked, since search engines need them to render your pages.
- Any crawler with its own group repeats the general rules it should still follow, because it will ignore the
*group. - The file opens at your domain followed by
/robots.txt, in plain text. - You have tested a few key paths, such as the home page and a service page, with the robots.txt tester.
Questions people ask
Where does robots.txt go?
In the root of your domain, so that it opens at https://www.yourdomain/robots.txt. Each subdomain needs its own file. On WordPress, many SEO plugins manage a virtual robots.txt for you; on other hosts, upload the downloaded file through the file manager or FTP into the site's top folder.
What does an empty Disallow mean?
A group with Disallow: and nothing after it allows everything for that crawler. The generator writes it automatically when a group has no paths at all, so the group is still valid. It is a common way to say a particular crawler may visit the whole site.
Should I add a crawl delay?
Google's documentation says it does not support the Crawl-delay line, though Bing reads it, so it rarely helps a small site. This generator does not offer it. If a crawler is putting real strain on your server, your host's bot or rate limiting settings are a more reliable fix.
Does robots.txt keep private pages secret?
No. The file is public, so listing a private folder in it tells everyone where that folder is. Anything genuinely private should sit behind a login or be removed from the server. Use robots.txt to steer well-behaved crawlers, not to protect information.
Where your data goes
The rules you type are assembled into a file locally and offered as a download. No part of your site structure is uploaded or recorded.
Checked against the tool on