Map Extract
Free tool

Robots.txt tester

Paste a robots.txt file and a list of paths or full addresses, then choose a crawler. Each path is marked allowed or blocked using the matching rules in RFC 9309, and any Sitemap lines in the file are listed underneath.

Runs in your browser. Nothing you enter is uploaded.

Testing a file

  1. Paste the robots.txt

    Open the domain followed by /robots.txt in your browser, copy the whole file and paste it. A draft you have not published yet works just as well.

  2. List what to test

    One path or URL per line. Full addresses are reduced to their path and query, so you can paste links straight from a spreadsheet.

  3. Choose the crawler

    Ten are offered: Googlebot, Bingbot, GPTBot, ClaudeBot, CCBot, Applebot, DuckDuckBot, YandexBot, AhrefsBot and SemrushBot.

  4. Read the table

    Every path is marked Allowed or Blocked for that crawler. Change the crawler to see how a named group changes the answer.

Example: two surprises in a roofer's file

A roofing firm (fictional) blocks its quote steps, allows the start page and tries to block PDFs. It also has a separate group for Bingbot. Testing the same six paths for two crawlers shows two things many people miss.

First, the $ in /*.pdf$ means the address must end there, so a PDF link with ?v=2 on the end is allowed. Second, Bingbot has its own group, so it ignores the * group entirely and may crawl the quote steps.

Input
User-agent: *
Disallow: /quote/
Allow: /quote/start
Disallow: /*.pdf$

User-agent: Bingbot
Disallow: /staff/

Sitemap: https://www.harbourroofing.example/sitemap.xml
Result
Path                             Googlebot   Bingbot
/                                Allowed     Allowed
/quote/step-2                    Blocked     Allowed
/quote/start                     Allowed     Allowed
/brochures/price-list.pdf        Blocked     Allowed
/brochures/price-list.pdf?v=2    Allowed     Allowed
/staff/rota                      Allowed     Blocked

The matching rules the tester follows

RuleWhat it means in practice
Groups that name the crawler winIf any group names it, only those groups apply and the * group is ignored
Several groups for one crawler combineRules from every group naming it are pooled
The longest matching rule winsAllow: /quote/start beats Disallow: /quote/ because it is longer
Allow wins a tieEqual-length Allow and Disallow rules let the path through
* matches anything, $ marks the end/*.pdf$ blocks addresses ending in .pdf, not ones with a query after it
No matching rule means allowedAn empty Disallow, or no rule for the path, blocks nothing

Why prospectors should know this too

If you collect business details from websites, robots.txt is the site owner's statement of what automated visitors may fetch. Reading it before you crawl is basic courtesy and part of showing you collected data responsibly. The tester lets you check a specific path, such as a contact page, for a given crawler name before you decide whether to visit it automatically.

Matching is by substring of the crawler name, so testing Googlebot also picks up a group written for googlebot in lower case.

Questions people ask

How do I test whether a URL is blocked by robots.txt?

Paste the site's robots.txt into this tester, put the URL or its path in the list, and choose the crawler you care about. The table shows whether that crawler is allowed to fetch it. Google Search Console also has a robots.txt report for sites you have verified there.

Does blocking a page in robots.txt remove it from Google?

Not necessarily. robots.txt stops crawling, not indexing: a blocked page can still appear in results, usually without a description, if other sites link to it. To keep a page out of search results, let it be crawled and add a noindex robots meta tag instead.

Why does a named crawler ignore my User-agent: * rules?

Under RFC 9309, a crawler uses only the groups that name it when such groups exist. As soon as you add a group for Bingbot, for example, Bingbot stops reading the asterisk group, so repeat any rules you still want it to follow inside its own group.

Does the tester fetch robots.txt from the website?

No. You paste the file yourself, which means it can test a draft before you publish it, and it never contacts the site. Open the live file in your browser, copy it in, and run the same paths again after any change to confirm the effect.

Where your data goes

Rules and paths are evaluated against each other in your browser using the page's own matcher. The robots.txt is not downloaded from the site and your paths are not logged.

Checked against the tool on

Top