Free tool
Free robots.txt tester
Enter a page URL and a user-agent. See whether robots.txt allows or blocks that path, and the exact line that decided it. No signup. Also see the sitemap checker.
How the tester decides
Enter a page URL, such as https://example.com/docs/page, and a product token such as Googlebot, Bingbot, GPTBot, or *. The tester requests /robots.txt on that same scheme, host, and port. Paste a file when you want to test a draft. Leave the box empty to fetch the live file.
Crawlers follow one group. The product token is compared with each User-agent line without regard to case. Google calls this the most specific match: a Googlebot-News group is for Googlebot-News, and a Googlebot group is for Googlebot. A different token does not inherit those rules. If several groups name the same token, their rules are combined. The * group is used only when the token is not listed, and its rules are not added to a specific group. A Sitemap line, or another record such as crawl-delay, does not start a new group. Only an Allow or Disallow line ends the run of User-agent lines.
The path, including the query string and not the fragment, is compared with the Allow and Disallow lines in that group. Matching is case-sensitive and starts at the first character of the path. * means any sequence of characters. $ means the end of the path. The longest matching rule wins. If an Allow and a Disallow are the same length, Allow wins. An empty Allow or Disallow is ignored. If nothing matches, the path is allowed. RFC 9309 also allows the /robots.txt URI itself.
robots.txt controls crawling, not indexing. Google says a URL blocked here can still be indexed when other pages link to it, usually without a snippet, because the page was not crawled. A noindex robots meta tag or HTTP header is what keeps a page out of Google, and Google has to be allowed to crawl the page to see it.
HTTP 404, and other 4xx responses except 429, mean the file is missing and every path is allowed. A 5xx response, a timeout, or a DNS failure means the file is unreachable. This tester does not keep a cache, so it does not invent an allow or a block from a file it could not read. Parsing stops after 500 KiB. Private hosts, localhost, and link-local addresses are refused, including after DNS and redirects. Redirects stop after three hops. The URL and any file you paste are not stored.
Questions
What does this robots.txt tester check?
It requests /robots.txt on the same scheme, host, and port as the page URL you enter, or it uses the file you paste. It then tests that page's path, including the query string, against the Allow and Disallow rules for the user-agent you pick.
Which user-agent group is used?
The group whose User-agent line matches the product token you pick, compared without regard to case. Google describes this as the most specific match: Googlebot-News follows a Googlebot-News group, and Googlebot follows a Googlebot group. A different token does not use those rules. If several groups name the same token, their rules are combined. The * group is used only when the token is not listed, and its rules are not added to a specific group.
How does it decide between Allow and Disallow?
The longest matching rule wins. Length is the length of the path in the rule, and * and $ count. If an Allow and a Disallow are the same length and both match, Allow wins. * matches any sequence of characters, including an empty one. $ matches the end of the path. An empty Allow or Disallow is ignored. If nothing matches, the path is allowed. RFC 9309 also allows the /robots.txt URI itself.
Does robots.txt keep a page out of Google?
No. It controls crawling, not indexing. Google says a URL blocked in robots.txt can still appear in search results when other pages link to it, usually without a snippet, because the page itself was not crawled. A noindex robots meta tag or HTTP header is what tells Google not to index a page, and Google has to be allowed to crawl the page to see it.
What if robots.txt is missing or the server errors?
A 404, and other 4xx responses except 429, mean there is no robots.txt. Google then crawls with no restrictions from that file. RFC 9309 says an unavailable robots.txt lets the crawler access the site. A 5xx response, a timeout, or a DNS failure means the file is unreachable. RFC 9309 says crawlers must assume complete disallow while it is unreachable. Google retries and, if it has no cached copy, crawls as if there were no file. This tester does not keep a cache, so it reports the file as unreachable instead of inventing an allow or a block.
Launching soon? List it on Sidehunt
Once the pages you want crawled are allowed, put the product in front of people who try new tools every week. Free, nofollow, and premium launches are listed on pricing.
AI-friendly Markdown · structured for AI citations