Write a robots.txt with AI crawler toggles
Choose allow or block for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest, add your paths and sitemap, and get a robots.txt with warnings for the common mistakes. Free, no account.
Your robots.txt
Example inputs, pre-filled so you can see how it works. Replace them with your own details.
A robots.txt of 24 lines: 6 of 14 listed crawlers blocked on /, 8 allowed.
- Pass: Checked by reading it back. We parsed the file we wrote and it blocks 6 of the 14 crawlers on /.
- Note: robots.txt is not security. The file is public and lists the paths you name. A blocked URL can still appear in results if others link to it; to keep a page out of Google use a noindex tag and leave it crawlable.
- Note: Google-Extended. Google's documentation says this token does not affect how a site appears in Google Search.
- Note: Blocking is not a deletion. A block asks crawlers to stop from now on. It does not remove anything already collected, and a crawler that ignores robots.txt is not stopped by it.
Put it live
Save it as robots.txt in the site root so it loads at https://yourdomain/robots.txt, then read it back with the AI crawler checker.
Only your choices are written. A robots.txt is a request, not a lock: well-behaved crawlers follow it, others do not. Whether to allow AI crawlers is the site owner's decision.
Common robots.txt mistakes
Disallow: /underUser-agent: *blocks the whole site, search engines included. Keep it for staging.- Blocking CSS, JavaScript or image folders can stop Google rendering pages the way visitors see them.
- Using robots.txt to hide pages: the file is public, and a blocked URL can still be listed. Use noindex for that.
- Forgetting that a named group replaces the
*group: a crawler with its own group ignores the rules under*. The generator repeats your rules in each allowed crawler's group for that reason. - A sitemap line that is not a full address.
Which crawlers to block
That is the owner's call. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent) collect text for models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) feed answers that can mention and link a site. Blocking the second group can keep a business out of those answers.
Find the businesses to sell to
Check each prospect's current file first, then offer the rewrite. Search 67Leads by trade and city, or start with businesses with no website and businesses with an owner email.
Get 67 leads free No card. Search free with contacts masked.
AI crawler checkerllms.txt generatorMeta tag checkerWebsite launch checklistAll lead lists67Leads pricing
Questions
Add a group with User-agent: GPTBot and Disallow: /. Set the crawler to Block here and the generator writes it.
It depends on the site. Blocking training crawlers keeps content out of future training; blocking search crawlers can keep the site out of AI answers. Neither choice is wrong in itself.
Google's documentation says it does not affect how a site appears in Google Search.
Under User-agent: * it asks every crawler to stay out of the whole site, so the site can drop out of search.
No. It writes the file from what you type and makes no network calls.
No. The result is written when the page is served and not saved.
More free tools
Local business counterNo-website business finderLead list previewCold email template generatorCold call objection handlerLocal SEO audit checklist generatorAgency client onboarding checklist generatorLocal citation list builderLocal keyword generatorNo-website pitch email generatorDiscovery call questions generatorGoogle Business Profile post generatorGatekeeper script generatorDoor-to-door sales script generatorAppointment setting script generatorSMS follow-up template generatorWebsite audit email generatorAgency profit margin calculatorLeads needed calculatorRetainer vs project pricing calculatorFacebook ad copy generator for local servicesLocal landing page outline generatorWebsite maintenance plan pricing calculatorReferral request script generatorCold email subject line generator for local businessesClient report template generatorGoogle Ads budget estimator for local tradesProposal follow-up email generatorWin-back email generator for lost clientsService pricing sheet generatorTrade website copy generatorAbout us page generator for local businessesTestimonial request message generatorCase study template generatorFreelancer and agency commission split calculatorClient churn rate calculator for agenciesTrade job price calculatorTrade estimate template generatorGrand opening press release generatorNextdoor post generator for local businessesYard sign and door hanger copy generatorCall tracking ROI calculatorMissed-call text-back message generatorLead response time cost calculatorAgency revenue goal plannerCold outreach volume plannerSales pipeline value calculatorAgency break-even calculatorClient price increase email generatorClient offboarding checklist generatorScope of work template generatorSales call follow-up email generatorCold email reply templates generatorWebsite launch checklist generatorClient kickoff meeting agenda generatorGoogle Business Profile Q&A generatorWebsite redesign ROI calculatorClient profitability calculatorAd management fee calculatorAgency utilization rate calculatorDental practice website pitch generatorMed spa offer post generatorLaw firm intake form checklist generatorAppointment no-show cost calculatorClinic website audit checklist generatorTrust signals section generatorMembership revenue calculatorAppointment reminder message generatorAI crawler checkerllms.txt generatorllms.txt checkerLocal business schema generatorFAQ schema generatorMeta tag checkerSchema markup checkerAll For web designers toolsAll tools