A robots.txt generator creates the plain-text file that tell search engines and AI crawlers which parts of your website they may visit. Choose a preset, list the paths to block, add your sitemap, set rules for bots like GPTBot, then copy or download the file. A built-in validator checks any existing file.
How to use this robots.txt generator
- Pick a preset. "Allow all crawlers" is a safe start. "Opt out of AI training, keep AI search" blocks nine training and data-collection bots. "Block everything" suits a staging site.
- Add your paths. Press a quick-path button for WordPress, WooCommerce or Shopify, or type your own Disallow and Allow paths, one per line.
- Paste your sitemap URL so crawlers can find your pages.
- Set AI crawlers. Switch any bot to Allow, Block or Inherit.
- Press Generate robots.txt. The preview updates as you edit. Fix any red message under a field.
- Download or copy the file and upload it to the root of your domain so it opens at https://yourdomain.com/robots.txt.
What a robots.txt file contains
The file is made of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers. A crawler follows only the single group that matches it best, so rules written for * do not apply to a bot that has its own group.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler a group applies to. * means every crawler. | User-agent: Googlebot |
| Disallow | Blocks a path prefix. An empty value blocks nothing. | Disallow: /admin/ |
| Allow | Re-opens a path inside a blocked folder. | Allow: /admin/help/ |
| Sitemap | Gives the full URL of an XML sitemap. | Sitemap: https://example.com/sitemap.xml |
| Crawl-delay | Asks a crawler to wait between requests. Google ignores it; some other engines respect it. | Crawl-delay: 10 |
Paths are case-sensitive and match from the start of the URL path. Two wildcards are widely supported: * matches any run of characters, and $ marks the end of a URL. Anything after # on a line is a comment.
How to block GPTBot and other AI crawlers
AI companies run several different bots, and each does a different job. Treat them separately:
- Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider collect pages that may be used to train models.
- Search crawlers such as OAI-SearchBot and PerplexityBot index pages so an AI answer can cite and link to you.
- User-initiated fetchers such as ChatGPT-User load a page when a person asks about it. Providers may handle these visits differently from automated crawling.
- Control tokens such as Google-Extended and Applebot-Extended are not separate crawlers. They tell Google and Apple whether your content may be used for AI training, and they do not affect Google Search or Apple's search crawler.
Blocking training bots while allowing search bots is a common compromise, and it is what the opt-out preset does. When you set a bot to Allow, this tool repeats your Disallow paths inside that bot's group. Otherwise the bot would ignore the * group and could enter folders such as /admin/.
Robots.txt examples with real numbers
A WordPress blog
Press the WordPress button, add the sitemap https://example.com/sitemap.xml and generate. The file blocks /wp-admin/, /wp-login.php, /?s= and /search/, and re-opens /wp-admin/admin-ajax.php with an Allow rule. For /wp-admin/admin-ajax.php both rules match, and the longer Allow rule wins, so it is allowed. The path /wp-admin/options.php matches only the Disallow rule, so it is blocked.
Opting out of AI training
Choose the training preset and keep the default /admin/ path. The result card shows 9 AI bots blocked, 3 inherit and a file of 21 lines: one comment line, 2 lines for the * group, and 2 lines for each of the 9 bot groups. Add a sitemap and the count becomes 22.
Wildcards and the longest-match rule
| Rules | URL path tested | Result |
|---|---|---|
| Disallow: /*.pdf$ | /files/report.pdf | Blocked |
| Disallow: /*.pdf$ | /files/report.pdf?v=2 | Allowed, because $ requires the URL to end in .pdf |
| Disallow: /private/ and Allow: /private/press/ | /private/press/kit.html | Allowed, because the longer rule wins |
| Disallow: /private/ and Allow: /private/press/ | /private/notes.html | Blocked |
How to validate and test a robots.txt file
Switch to Validate, paste your file and press Check. The tool reads it line by line and reports three kinds of findings:
- Errors such as a rule placed before any User-agent line, a line that is not a valid directive, or a Sitemap that is not a full URL.
- Warnings such as unsupported directives (for example Host or noindex), paths that do not start with /, or Disallow: / for all crawlers.
- Tips such as a missing Sitemap line, or the fact that Google ignores Crawl-delay.
You can also test one URL. Enter a user-agent such as Googlebot and a path such as /private/notes.html. The tool finds the matching group, applies the longest-match rule and shows which line allowed or blocked the path. The AI bot table runs the same check for twelve AI crawlers at the root path.
Common mistakes and honest limits
- robots.txt is not security. It is a request that well-behaved crawlers follow. Protect private pages with a login, not a Disallow line.
- Blocking is not removing. A blocked URL can still appear in search results if other sites link to it. Use a noindex tag on a crawlable page to keep it out of the index.
- Do not block CSS and JavaScript that your pages need to render, because search engines use them to understand your layout.
- One file per host. A subdomain such as blog.example.com needs its own robots.txt.
- Disallow: / hides everything. Use it on a staging site and remove it before launch.
- Bot names change. The list here covers well-known AI crawlers today. New ones appear, and some scrapers ignore robots.txt completely.
- Keep the file small. Google reads only the first 500 KiB.
Everything on this page runs in your browser, and nothing you type is uploaded.
Who uses a robots.txt generator
- Site owners keep admin, cart and search pages out of crawl budgets.
- Developers block staging sites and add the right file at launch.
- SEO specialists audit client files and test URLs before a release.
- Publishers and writers decide which AI bots may use their content.
Looking for more utilities? Browse our other free online tools for site owners, developers and writers.