# Free Robots.txt Tester: Is Your Page Blocked? | Sidehunt
> Paste a page URL and a user-agent. See if robots.txt allows or blocks that path, which line decided it, and any Sitemap lines. Free, no signup.

```yaml
url: "https://sidehunt.io/tools/robots-txt-tester"
markdown: "https://sidehunt.io/tools/robots-txt-tester.md"
title: "Free Robots.txt Tester: Is Your Page Blocked? | Sidehunt"
description: "Paste a page URL and a user-agent. See if robots.txt allows or blocks that path, which line decided it, and any Sitemap lines. Free, no signup."
site: Sidehunt
```

## Overview

Paste a page URL and a user-agent. See if robots.txt allows or blocks that path, which line decided it, and any Sitemap lines. Free, no signup.

Enter a page URL and a product token such as Googlebot, or paste robots.txt. The checker requests /robots.txt on that scheme, host, and port, chooses the matching user-agent group, and reports whether the path is allowed or blocked, including the line that decided it. Sitemap lines are listed too. A 404 means the path is allowed. An unreachable file is reported as unreachable.

## FAQ

### What does this robots.txt tester check?

It requests /robots.txt on the same scheme, host, and port as the page URL you enter, or it uses the file you paste. It then tests that page's path, including the query string, against the Allow and Disallow rules for the user-agent you pick.

### Which user-agent group is used?

The group whose User-agent line matches the product token you pick, compared without regard to case. Google describes this as the most specific match: Googlebot-News follows a Googlebot-News group, and Googlebot follows a Googlebot group. A different token does not use those rules. If several groups name the same token, their rules are combined. The * group is used only when the token is not listed, and its rules are not added to a specific group.

### How does it decide between Allow and Disallow?

The longest matching rule wins. Length is the length of the path in the rule, and * and $ count. If an Allow and a Disallow are the same length and both match, Allow wins. * matches any sequence of characters, including an empty one. $ matches the end of the path. An empty Allow or Disallow is ignored. If nothing matches, the path is allowed. RFC 9309 also allows the /robots.txt URI itself.

### Does robots.txt keep a page out of Google?

No. It controls crawling, not indexing. Google says a URL blocked in robots.txt can still appear in search results when other pages link to it, usually without a snippet, because the page itself was not crawled. A noindex robots meta tag or HTTP header is what tells Google not to index a page, and Google has to be allowed to crawl the page to see it.

### What if robots.txt is missing or the server errors?

A 404, and other 4xx responses except 429, mean there is no robots.txt. Google then crawls with no restrictions from that file. RFC 9309 says an unavailable robots.txt lets the crawler access the site. A 5xx response, a timeout, or a DNS failure means the file is unreachable. RFC 9309 says crawlers must assume complete disallow while it is unreachable. Google retries and, if it has no cached copy, crawls as if there were no file. This tester does not keep a cache, so it reports the file as unreachable instead of inventing an allow or a block.


## Related

- [Sitemap checker](https://sidehunt.io/tools/sitemap-checker) — [Markdown](https://sidehunt.io/tools/sitemap-checker.md)
- [Submit a project](https://sidehunt.io/submit) — [Markdown](https://sidehunt.io/submit.md)
- [Pricing](https://sidehunt.io/pricing) — [Markdown](https://sidehunt.io/pricing.md)

## Links

- [HTML version](https://sidehunt.io/tools/robots-txt-tester)
- [AI-friendly Markdown](https://sidehunt.io/tools/robots-txt-tester.md)
- [llms.txt index](https://sidehunt.io/llms.txt)

_HTML version: https://sidehunt.io/tools/robots-txt-tester_