Testing a file
Paste the robots.txt
Open the domain followed by
/robots.txtin your browser, copy the whole file and paste it. A draft you have not published yet works just as well.List what to test
One path or URL per line. Full addresses are reduced to their path and query, so you can paste links straight from a spreadsheet.
Choose the crawler
Ten are offered: Googlebot, Bingbot, GPTBot, ClaudeBot, CCBot, Applebot, DuckDuckBot, YandexBot, AhrefsBot and SemrushBot.
Read the table
Every path is marked Allowed or Blocked for that crawler. Change the crawler to see how a named group changes the answer.
Example: two surprises in a roofer's file
A roofing firm (fictional) blocks its quote steps, allows the start page and tries to block PDFs. It also has a separate group for Bingbot. Testing the same six paths for two crawlers shows two things many people miss.
First, the $ in /*.pdf$ means the address must end there, so a PDF link with ?v=2 on the end is allowed. Second, Bingbot has its own group, so it ignores the * group entirely and may crawl the quote steps.
User-agent: * Disallow: /quote/ Allow: /quote/start Disallow: /*.pdf$ User-agent: Bingbot Disallow: /staff/ Sitemap: https://www.harbourroofing.example/sitemap.xml
Path Googlebot Bingbot / Allowed Allowed /quote/step-2 Blocked Allowed /quote/start Allowed Allowed /brochures/price-list.pdf Blocked Allowed /brochures/price-list.pdf?v=2 Allowed Allowed /staff/rota Allowed Blocked
The matching rules the tester follows
| Rule | What it means in practice |
|---|---|
| Groups that name the crawler win | If any group names it, only those groups apply and the * group is ignored |
| Several groups for one crawler combine | Rules from every group naming it are pooled |
| The longest matching rule wins | Allow: /quote/start beats Disallow: /quote/ because it is longer |
| Allow wins a tie | Equal-length Allow and Disallow rules let the path through |
* matches anything, $ marks the end | /*.pdf$ blocks addresses ending in .pdf, not ones with a query after it |
| No matching rule means allowed | An empty Disallow, or no rule for the path, blocks nothing |
Why prospectors should know this too
If you collect business details from websites, robots.txt is the site owner's statement of what automated visitors may fetch. Reading it before you crawl is basic courtesy and part of showing you collected data responsibly. The tester lets you check a specific path, such as a contact page, for a given crawler name before you decide whether to visit it automatically.
Matching is by substring of the crawler name, so testing Googlebot also picks up a group written for googlebot in lower case.
Questions people ask
How do I test whether a URL is blocked by robots.txt?
Paste the site's robots.txt into this tester, put the URL or its path in the list, and choose the crawler you care about. The table shows whether that crawler is allowed to fetch it. Google Search Console also has a robots.txt report for sites you have verified there.
Does blocking a page in robots.txt remove it from Google?
Not necessarily. robots.txt stops crawling, not indexing: a blocked page can still appear in results, usually without a description, if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex robots meta tag instead.
Why does a named crawler ignore my User-agent: * rules?
Under RFC 9309, a crawler uses only the groups that name it when such groups exist. As soon as you add a group for Bingbot, for example, Bingbot stops reading the asterisk group, so repeat any rules you still want it to follow inside its own group.
Does the tester fetch robots.txt from the website?
No. You paste the file yourself, which means it can test a draft before you publish it, and it never contacts the site. Open the live file in your browser, copy it in, and run the same paths again after any change to confirm the effect.
Where your data goes
Rules and paths are evaluated against each other in your browser using the page's own matcher. The robots.txt is not downloaded from the site and your paths are not logged.
Checked against the tool on