Check which AI crawlers a website allows
Type a website. We read its robots.txt and show, for 14 crawlers including GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Googlebot, whether each is allowed or blocked, and the rule that decides it. Free, no account.
Example: a sample page we wrote for this demo (smithplumbing.example is not a real site). Type any website above for a live check.
smithplumbing.example: AI crawler access
smithplumbing.example: 3 of 14 crawlers are blocked on /; 11 are allowed.
| Crawler | Operator and use | Result | Deciding rule |
|---|---|---|---|
| GPTBot | OpenAI: model training crawler | Blocked | Disallow: /uses its own group |
| OAI-SearchBot | OpenAI: search crawler for ChatGPT search results | Allowed | Group matches, no rule covers this pathuses the * group |
| ChatGPT-User | OpenAI: fetches a page when a ChatGPT user asks for it | Allowed | Group matches, no rule covers this pathuses the * group |
| ClaudeBot | Anthropic: model training crawler | Blocked | Disallow: /uses its own group |
| Claude-SearchBot | Anthropic: search crawler for Claude answers | Allowed | Group matches, no rule covers this pathuses the * group |
| Claude-User | Anthropic: fetches a page when a Claude user asks for it | Allowed | Group matches, no rule covers this pathuses the * group |
| PerplexityBot | Perplexity: search index crawler | Allowed | Group matches, no rule covers this pathuses the * group |
| Perplexity-User | Perplexity: fetches a page when a Perplexity user asks for it | Allowed | Group matches, no rule covers this pathuses the * group |
| Google-Extended | Google: control token for Gemini training and grounding; not a separate crawler | Allowed | Group matches, no rule covers this pathuses the * group |
| Googlebot | Google: Google Search crawler | Allowed | Group matches, no rule covers this pathuses the * group |
| Bingbot | Microsoft: Bing Search crawler | Allowed | Group matches, no rule covers this pathuses the * group |
| Applebot-Extended | Apple: control token for Apple AI training; not a separate crawler | Allowed | Group matches, no rule covers this pathuses the * group |
| CCBot | Common Crawl: open web archive crawler, used by many AI datasets | Blocked | Disallow: /uses its own group |
| meta-externalagent | Meta: crawler for Meta AI training and indexing | Allowed | Group matches, no rule covers this pathuses the * group |
The file we read
Verdicts follow the robots.txt standard (RFC 9309): a crawler uses the group that names it, else the * group; the longest matching rule wins and Allow wins a tie. Some operators say their user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User) may not follow robots.txt, and Google-Extended and Applebot-Extended are control tokens, not crawlers you will see in logs. Check each operator's own page for current behaviour.
A robots.txt asks crawlers to behave; it does not enforce anything. This is a read of the file, not a test of what any crawler actually does.
How the checker works
- We request
/robots.txton the site you type: http and https only, standard ports only, no IP addresses or internal names, every resolved address must be public, and redirects are followed by hand and re-checked. - The file is parsed the way the robots.txt standard describes: groups of User-agent lines, the group that names the
crawler or else the
*group, wildcards (*and$), the longest matching rule wins and Allow wins a tie. - A missing file (a 4xx answer) means everything is allowed. A server error (5xx) means we cannot say, and Google may treat the site as closed until it is fixed.
- It tests one path,
/unless you enter another, for 14 named crawlers. It does not test the pages' own noindex tags or any login wall.
Why web designers check this
Many sites block AI crawlers without knowing it: a plugin, a host default or an old copy-pasted file. Whether to allow them is the owner's decision. Showing a prospect exactly which crawlers their file blocks, and the line that does it, is a concrete, checkable thing to bring to a conversation.
Find the businesses to sell to
Check a prospect's robots.txt, then offer to fix it. Search 67Leads by trade and city, or start with businesses with no website and businesses with an owner email.
Get 67 leads free No card. Search free with contacts masked.
robots.txt generatorllms.txt generatorMeta tag checkerWebsite audit reportAll lead lists67Leads pricing
Questions
Type the site here. The table shows each crawler as allowed or blocked and the rule that decided it.
OpenAI lists them as separate crawlers: GPTBot is for training, OAI-SearchBot for search results in ChatGPT. A site can allow one and block the other. Check OpenAI's documentation for the current description.
Control tokens. They are written in robots.txt to set what Google and Apple may use for AI, but they do not crawl by themselves, so you will not see them in server logs.
No. robots.txt is a request that well-behaved crawlers follow. It does not remove data already collected.
No. The page is read when you ask and the result is written to the page, not saved.
No. Every address is resolved first and refused if it points at a private, loopback, link-local or reserved network, and every redirect is checked the same way.
More free tools
Local business counterNo-website business finderLead list previewCold email template generatorCold call objection handlerLocal SEO audit checklist generatorAgency client onboarding checklist generatorLocal citation list builderLocal keyword generatorNo-website pitch email generatorDiscovery call questions generatorGoogle Business Profile post generatorGatekeeper script generatorDoor-to-door sales script generatorAppointment setting script generatorSMS follow-up template generatorWebsite audit email generatorAgency profit margin calculatorLeads needed calculatorRetainer vs project pricing calculatorFacebook ad copy generator for local servicesLocal landing page outline generatorWebsite maintenance plan pricing calculatorReferral request script generatorCold email subject line generator for local businessesClient report template generatorGoogle Ads budget estimator for local tradesProposal follow-up email generatorWin-back email generator for lost clientsService pricing sheet generatorTrade website copy generatorAbout us page generator for local businessesTestimonial request message generatorCase study template generatorFreelancer and agency commission split calculatorClient churn rate calculator for agenciesTrade job price calculatorTrade estimate template generatorGrand opening press release generatorNextdoor post generator for local businessesYard sign and door hanger copy generatorCall tracking ROI calculatorMissed-call text-back message generatorLead response time cost calculatorAgency revenue goal plannerCold outreach volume plannerSales pipeline value calculatorAgency break-even calculatorClient price increase email generatorClient offboarding checklist generatorScope of work template generatorSales call follow-up email generatorCold email reply templates generatorWebsite launch checklist generatorClient kickoff meeting agenda generatorGoogle Business Profile Q&A generatorWebsite redesign ROI calculatorClient profitability calculatorAd management fee calculatorAgency utilization rate calculatorDental practice website pitch generatorMed spa offer post generatorLaw firm intake form checklist generatorAppointment no-show cost calculatorClinic website audit checklist generatorTrust signals section generatorMembership revenue calculatorAppointment reminder message generatorllms.txt generatorllms.txt checkerLocal business schema generatorFAQ schema generatorrobots.txt generatorMeta tag checkerSchema markup checkerAll Website checks toolsAll tools