Check whether a page can be indexed
Type a page address. We check its status code, redirects, meta robots, X-Robots-Tag, canonical, robots.txt and sitemap in one pass and say whether anything tells search engines to stay away. Free, no account.
How the check works
- The address is parsed first: http and https only, ports 80 and 443 only, no usernames, no IP addresses and no internal names such as localhost. Anything else is refused before any request.
- The name is resolved and every address it points to must be public. Private, loopback, link-local (cloud metadata included), multicast and reserved addresses are refused, and the connection goes to the address that was checked.
- Addresses found inside the page, such as icons, a manifest, a contact page or a sitemap, pass the same checks before they are requested; one that points at a private address is refused, not fetched.
- We request the page, following up to 4 redirects, then the site's /robots.txt, then up to 3 sitemap files (from robots.txt Sitemap lines, else /sitemap.xml). Everything shares one 20-second budget.
- robots.txt is tested for Googlebot with the RFC 9309 rules our robots.txt tester uses.
What can block indexing
- An error status: 4xx or 5xx answers are dropped from search results.
- noindex in a robots or googlebot meta tag, or in an X-Robots-Tag header (which also covers PDFs and images).
- robots.txt Disallow: the crawler cannot read the page. Note that it then never sees a noindex tag either.
- A robots.txt server error: Google stops crawling the site while robots.txt answers 5xx.
A canonical pointing elsewhere and a missing sitemap entry are not blocks, so they show as things to check.
Why it helps before an audit
A stray noindex left on from a site rebuild is one of the most common reasons a small business gets no search traffic, and it takes a minute to fix. To find businesses to check, search 67Leads by trade and city.
Get 67 leads free No card. Search free with contacts masked.
Robots.txt testerCanonical tag checkerSitemap validatorRedirect checkerMeta tag checker
Questions
Start with this check. If nothing blocks it, the page may be new, have no links pointing at it, or be seen as a duplicate. Search Console's URL Inspection tool says what Google decided.
noindex lets the crawler read the page but asks it not to list it. Disallow stops the crawler reading the page at all, so it never sees a noindex tag; the bare address can still appear if other sites link to it.
Not directly. It tells search engines which address you prefer, and they usually index that one instead. It is a hint, not a rule.
No. A sitemap helps search engines find pages, but pages linked from elsewhere get indexed without it.
No. Our server fetches the address you enter, its robots.txt and up to three sitemap files, reads the robots signals and reports what it found. No AI model is involved, and the address and result are not stored.
More free tools
Local business counterNo-website business finderLead list previewCold email template generatorAI crawler checkerllms.txt generatorllms.txt checkerLocal business schema generatorFAQ schema generatorrobots.txt generatorMeta tag checkerSchema markup checkerNAICS code lookupSIC code lookupSSL certificate checkerRedirect checkerBroken link checkerPage weight checkerLead list column mapperMerge tag checkerOpen Graph checkerCanonical tag checkerSitemap validatorHTTP headers checkerHeading structure checkerImage alt text checkerHreflang checkerDNS lookupAI Overview readiness checkerRobots.txt testerWebsite speed testMixed content checkerFavicon checkerSocial profile finderNofollow link checkerAll Website checks toolsAll tools