Test a path against a site's robots.txt
Type a full page address. We read the site's robots.txt and test that path for the crawlers you tick, showing the exact rule that decides each one. Free, no account.
How the test works
- The address is parsed first: http and https only, ports 80 and 443 only, no usernames, no IP addresses and no internal names such as localhost. Anything else is refused before any request.
- The name is resolved and every address it points to must be public. Private, loopback, link-local (cloud metadata included), multicast and reserved addresses are refused, and the connection goes to the address that was checked.
- We read /robots.txt on the host you typed (512 KB at most, the size Google reads) and take the path and query from your address. With no path, we test /.
- For each crawler we pick every group that names it, or the * group if none does, list the rules whose path matches and pick the longest. On a tie Allow wins. A * in a rule matches anything and a $ at the end means the path must end there.
- A missing robots.txt (4xx) allows everything; a server error (5xx) is read as "disallow everything", as the standard says.
Reading the result
The deciding rule is the line that settled the answer. "Also matched" lists shorter rules that matched but lost. If a crawler shows "no rule matches", the group applies but says nothing about this path, so the path is allowed.
A rule that blocks /wp-content/ or /assets/ can stop Google rendering a page properly, and a blanket Disallow: / left from a staging site blocks everything. Both are quick fixes worth raising with a site owner; to find businesses to check, search 67Leads by trade and city.
Get 67 leads free No card. Search free with contacts masked.
AI crawler checkerrobots.txt generatorIndexability checkerSitemap validatorAI Overview readiness checker
Questions
The one with the longer path. If both are the same length, Allow wins. That is the rule in RFC 9309, which Google and Bing follow.
* matches any run of characters, so Disallow: /*.pdf$ matches every address ending in .pdf. $ at the end means the address must end there; without it a rule matches anything that starts with the path.
No. A crawler that finds a group naming it uses only that group (or several, if more than one names it) and ignores the * group.
Not reliably. It stops crawling, but the bare address can still be listed if other pages link to it. Use a noindex tag on a crawlable page to keep it out.
No. Our server fetches the robots.txt of the address you enter, applies its rules to your path and reports what it found. No AI model is involved, and the address and result are not stored.
More free tools
Local business counterNo-website business finderLead list previewCold email template generatorAI crawler checkerllms.txt generatorllms.txt checkerLocal business schema generatorFAQ schema generatorrobots.txt generatorMeta tag checkerSchema markup checkerNAICS code lookupSIC code lookupSSL certificate checkerRedirect checkerBroken link checkerPage weight checkerLead list column mapperMerge tag checkerOpen Graph checkerCanonical tag checkerSitemap validatorHTTP headers checkerHeading structure checkerImage alt text checkerHreflang checkerDNS lookupAI Overview readiness checkerIndexability checkerWebsite speed testMixed content checkerFavicon checkerSocial profile finderNofollow link checkerAll Website checks toolsAll tools