Developer Tools

Robots.txt Generator - Free Online Tool & Validator

Free robots.txt generator that builds crawl rules, sitemap lines and AI bot blocks in your browser, and checks any existing file for errors.

Published by Free Online Tools Last updated: 5 uses

What do you want to do?

Rules

Quick paths

AI crawlers

Inherit means no separate rule, so the bot follows the * group.

No robots.txt yet

Choose your rules and press Generate robots.txt.

A robots.txt generator creates the plain-text file that tell search engines and AI crawlers which parts of your website they may visit. Choose a preset, list the paths to block, add your sitemap, set rules for bots like GPTBot, then copy or download the file. A built-in validator checks any existing file.

How to use this robots.txt generator

  1. Pick a preset. "Allow all crawlers" is a safe start. "Opt out of AI training, keep AI search" blocks nine training and data-collection bots. "Block everything" suits a staging site.
  2. Add your paths. Press a quick-path button for WordPress, WooCommerce or Shopify, or type your own Disallow and Allow paths, one per line.
  3. Paste your sitemap URL so crawlers can find your pages.
  4. Set AI crawlers. Switch any bot to Allow, Block or Inherit.
  5. Press Generate robots.txt. The preview updates as you edit. Fix any red message under a field.
  6. Download or copy the file and upload it to the root of your domain so it opens at https://yourdomain.com/robots.txt.

What a robots.txt file contains

The file is made of groups. Each group starts with one or more User-agent lines, followed by the rules for those crawlers. A crawler follows only the single group that matches it best, so rules written for * do not apply to a bot that has its own group.

DirectiveWhat it doesExample
User-agentNames the crawler a group applies to. * means every crawler.User-agent: Googlebot
DisallowBlocks a path prefix. An empty value blocks nothing.Disallow: /admin/
AllowRe-opens a path inside a blocked folder.Allow: /admin/help/
SitemapGives the full URL of an XML sitemap.Sitemap: https://example.com/sitemap.xml
Crawl-delayAsks a crawler to wait between requests. Google ignores it; some other engines respect it.Crawl-delay: 10

Paths are case-sensitive and match from the start of the URL path. Two wildcards are widely supported: * matches any run of characters, and $ marks the end of a URL. Anything after # on a line is a comment.

How to block GPTBot and other AI crawlers

AI companies run several different bots, and each does a different job. Treat them separately:

  • Training crawlers such as GPTBot, ClaudeBot, CCBot and Bytespider collect pages that may be used to train models.
  • Search crawlers such as OAI-SearchBot and PerplexityBot index pages so an AI answer can cite and link to you.
  • User-initiated fetchers such as ChatGPT-User load a page when a person asks about it. Providers may handle these visits differently from automated crawling.
  • Control tokens such as Google-Extended and Applebot-Extended are not separate crawlers. They tell Google and Apple whether your content may be used for AI training, and they do not affect Google Search or Apple's search crawler.

Blocking training bots while allowing search bots is a common compromise, and it is what the opt-out preset does. When you set a bot to Allow, this tool repeats your Disallow paths inside that bot's group. Otherwise the bot would ignore the * group and could enter folders such as /admin/.

Robots.txt examples with real numbers

A WordPress blog

Press the WordPress button, add the sitemap https://example.com/sitemap.xml and generate. The file blocks /wp-admin/, /wp-login.php, /?s= and /search/, and re-opens /wp-admin/admin-ajax.php with an Allow rule. For /wp-admin/admin-ajax.php both rules match, and the longer Allow rule wins, so it is allowed. The path /wp-admin/options.php matches only the Disallow rule, so it is blocked.

Opting out of AI training

Choose the training preset and keep the default /admin/ path. The result card shows 9 AI bots blocked, 3 inherit and a file of 21 lines: one comment line, 2 lines for the * group, and 2 lines for each of the 9 bot groups. Add a sitemap and the count becomes 22.

Wildcards and the longest-match rule

RulesURL path testedResult
Disallow: /*.pdf$/files/report.pdfBlocked
Disallow: /*.pdf$/files/report.pdf?v=2Allowed, because $ requires the URL to end in .pdf
Disallow: /private/ and Allow: /private/press//private/press/kit.htmlAllowed, because the longer rule wins
Disallow: /private/ and Allow: /private/press//private/notes.htmlBlocked

How to validate and test a robots.txt file

Switch to Validate, paste your file and press Check. The tool reads it line by line and reports three kinds of findings:

  • Errors such as a rule placed before any User-agent line, a line that is not a valid directive, or a Sitemap that is not a full URL.
  • Warnings such as unsupported directives (for example Host or noindex), paths that do not start with /, or Disallow: / for all crawlers.
  • Tips such as a missing Sitemap line, or the fact that Google ignores Crawl-delay.

You can also test one URL. Enter a user-agent such as Googlebot and a path such as /private/notes.html. The tool finds the matching group, applies the longest-match rule and shows which line allowed or blocked the path. The AI bot table runs the same check for twelve AI crawlers at the root path.

Common mistakes and honest limits

  • robots.txt is not security. It is a request that well-behaved crawlers follow. Protect private pages with a login, not a Disallow line.
  • Blocking is not removing. A blocked URL can still appear in search results if other sites link to it. Use a noindex tag on a crawlable page to keep it out of the index.
  • Do not block CSS and JavaScript that your pages need to render, because search engines use them to understand your layout.
  • One file per host. A subdomain such as blog.example.com needs its own robots.txt.
  • Disallow: / hides everything. Use it on a staging site and remove it before launch.
  • Bot names change. The list here covers well-known AI crawlers today. New ones appear, and some scrapers ignore robots.txt completely.
  • Keep the file small. Google reads only the first 500 KiB.

Everything on this page runs in your browser, and nothing you type is uploaded.

Who uses a robots.txt generator

  • Site owners keep admin, cart and search pages out of crawl budgets.
  • Developers block staging sites and add the right file at launch.
  • SEO specialists audit client files and test URLs before a release.
  • Publishers and writers decide which AI bots may use their content.

Looking for more utilities? Browse our other free online tools for site owners, developers and writers.

Frequently asked questions

Where do I put the robots.txt file?
Place it in the root folder of your domain so it loads at https://yourdomain.com/robots.txt. Crawlers check only that exact address. Each subdomain and each protocol needs its own file, and the file name must be lowercase.
Does robots.txt remove a page from Google?
No. Robots.txt only controls crawling, so a blocked URL can still be indexed if other pages link to it, usually without a description. To keep a page out of search results, leave it crawlable and add a noindex meta tag or header.
How do I block ChatGPT from my website?
Block GPTBot to opt out of OpenAI model training, and OAI-SearchBot to stay out of ChatGPT search results. ChatGPT-User fetches pages when a person asks, and OpenAI may treat those visits differently from crawling. Set each bot to Block in this tool.
Will blocking GPTBot hurt my Google rankings?
No. GPTBot belongs to OpenAI and has nothing to do with Googlebot, so blocking it does not affect Google Search. Google-Extended is also separate and only controls whether Google may use your content for Gemini training.
Is robots.txt a security feature?
No. It is a public file that polite crawlers choose to obey, and anyone can read it. Listing a folder under Disallow can even point people to it. Protect sensitive content with passwords or server rules instead.
What does an empty Disallow line mean?
An empty Disallow value means nothing is blocked, so the crawler may visit every page. It is the same as having no restriction. This tool writes Disallow: with no value under User-agent: * when you add no paths.
Can I use wildcards in robots.txt?
Yes. Major search engines support * for any run of characters and $ for the end of a URL. For example, Disallow: /*.pdf$ blocks URLs that end in .pdf. Smaller crawlers may not understand wildcards, so test important rules.
Do all AI crawlers obey robots.txt?
No. Reputable companies say their crawlers respect robots.txt, but the file cannot force anything, and some scrapers ignore it. For stronger protection, add firewall rules, rate limits or bot-management tools alongside robots.txt.
Where do I put the robots.txt file?
Place it in the root folder of your domain so it loads at https://yourdomain.com/robots.txt. Crawlers check only that exact address. Each subdomain and each protocol needs its own file, and the file name must be lowercase.