Test a path against a site's robots.txt

Type a full page address. We read the site's robots.txt and test that path for the crawlers you tick, showing the exact rule that decides each one. Free, no account.

How the test works

Reading the result

The deciding rule is the line that settled the answer. "Also matched" lists shorter rules that matched but lost. If a crawler shows "no rule matches", the group applies but says nothing about this path, so the path is allowed.

A rule that blocks /wp-content/ or /assets/ can stop Google rendering a page properly, and a blanket Disallow: / left from a staging site blocks everything. Both are quick fixes worth raising with a site owner; to find businesses to check, search 67Leads by trade and city.

Get 67 leads free No card. Search free with contacts masked.

AI crawler checkerrobots.txt generatorIndexability checkerSitemap validatorAI Overview readiness checker

Questions

Which rule wins when Allow and Disallow both match?

The one with the longer path. If both are the same length, Allow wins. That is the rule in RFC 9309, which Google and Bing follow.

What do * and $ mean in robots.txt?

* matches any run of characters, so Disallow: /*.pdf$ matches every address ending in .pdf. $ at the end means the address must end there; without it a rule matches anything that starts with the path.

If Googlebot has its own group, does it also read the * group?

No. A crawler that finds a group naming it uses only that group (or several, if more than one names it) and ignores the * group.

Does blocking a page in robots.txt remove it from Google?

Not reliably. It stops crawling, but the bare address can still be listed if other pages link to it. Use a noindex tag on a crawlable page to keep it out.

Does this use AI?

No. Our server fetches the robots.txt of the address you enter, applies its rules to your path and reports what it found. No AI model is involved, and the address and result are not stored.

More free tools

Local business counterNo-website business finderLead list previewCold email template generatorAI crawler checkerllms.txt generatorllms.txt checkerLocal business schema generatorFAQ schema generatorrobots.txt generatorMeta tag checkerSchema markup checkerNAICS code lookupSIC code lookupSSL certificate checkerRedirect checkerBroken link checkerPage weight checkerLead list column mapperMerge tag checkerOpen Graph checkerCanonical tag checkerSitemap validatorHTTP headers checkerHeading structure checkerImage alt text checkerHreflang checkerDNS lookupAI Overview readiness checkerIndexability checkerWebsite speed testMixed content checkerFavicon checkerSocial profile finderNofollow link checkerAll Website checks toolsAll tools