Robots.txt Generator

A robots.txt file is a plain text file placed at the root of a website — at yourdomain.com/robots.txt — that tells search engine crawlers and other automated bots which parts of the site they're allowed to request. It's read before a bot crawls anything else, so a single misplaced line can either leave sensitive areas exposed to crawling or accidentally block search engines from your entire site.

This tool generates a valid robots.txt file through three modes: Allow All (a simple file permitting every crawler everywhere), Block All (disallowing every crawler from every page), and Custom Rules, where you build one or more rules by specifying a user-agent along with allow and disallow paths. Quick-start templates for WordPress, Next.js, and Shopify fill in a sensible starting point for those platforms, which you can then edit directly.

Once your rules are set, the tool shows a live preview of the exact file content, with buttons to copy it to your clipboard or download it directly as robots.txt, ready to upload to your site's root directory.

Every robots.txt file is built from a small set of directives. User-agent names which crawler a rule applies to — an asterisk means the rule applies to all bots, while a specific name like Googlebot targets only that one. Disallow tells that user-agent not to crawl a given path, and Allow explicitly permits a path, which matters most when it carves out an exception inside a broader Disallow rule.

Sitemap is different from the other three: rather than controlling access, it simply points crawlers to your sitemap file's full URL so they can discover your pages more efficiently. This tool includes a dedicated Sitemap field in Allow All and Block All modes, and a per-rule Sitemap field in Custom Rules mode, and it always writes the URL out in full rather than as a relative path, since that's what the directive requires.

One directive this tool doesn't generate a dedicated field for is Crawl-delay, which some crawlers use to space out how frequently they request pages. If you need it, add a line reading Crawl-delay: followed by a number of seconds directly into the Custom Rules text after generating your other rules, positioned under the relevant User-agent line.

• User-agent: * — applies the rule to every crawler
• Disallow: /admin/ — blocks crawling of that specific path and everything under it
• Allow: /public/ — explicitly permits a path, often as an exception inside a Disallow block
• Sitemap: https://example.com/sitemap.xml — points crawlers to your sitemap file

The single most damaging robots.txt mistake is a line reading Disallow: / under User-agent: *, since that one slash means "everything," blocking every page on the entire site from every crawler that respects the file. This tool's Block All mode generates exactly that line on purpose, which is useful for a staging environment or a site not yet ready for search engines, but it should never be left in place on a live production site.

A second common mistake is disallowing a path that unintentionally covers pages you actually want indexed — for example, disallowing /blog when you meant to disallow only /blog/drafts. Because Disallow rules match anything starting with that path, it's worth double-checking each rule against your actual site structure before uploading, especially when editing a template rather than writing rules from scratch.

Beyond search engines, a growing number of bots crawl the web specifically to gather training data for AI systems — GPTBot (OpenAI) and CCBot (Common Crawl, used by several AI companies) are two of the most common. If you'd rather these bots not access your site's content, you can block them using the same Disallow mechanism as any other crawler, just targeted at their specific user-agent name instead of the wildcard asterisk.

In this tool's Custom Rules mode, add a separate rule with the user-agent set to GPTBot and Disallow set to /, then repeat with a new rule for CCBot and any other bot name you want to block; each bot needs its own User-agent line since a single rule can't target multiple named bots at once. Note that robots.txt is a voluntary standard — well-behaved crawlers respect it, but it can't technically force a bot to comply if it chooses to ignore the file.

• GPTBot — used by OpenAI for crawling content
• CCBot — used by Common Crawl, whose data feeds multiple AI companies
• Add one User-agent rule per bot name you want to block; a shared wildcard rule won't target them individually

It's a common misunderstanding that disallowing a page in robots.txt keeps it out of search results entirely — it doesn't. Disallow only tells a crawler not to fetch the page's content; if search engines discover the URL some other way, such as a link from another site, they can still list the bare URL in search results without a title or description pulled from the page.

To reliably keep a specific page out of search results, use a noindex meta tag or HTTP header on that page itself instead of, or in addition to, a robots.txt rule — and note that for noindex to be seen at all, the page generally needs to remain crawlable, since a page blocked from crawling can't have its noindex tag read either. This tool generates only robots.txt content; a noindex tag needs to be added separately in your page's HTML head or server response.

Clicking a quick template switches the tool into Custom Rules mode and fills the editable text area with a starting file tailored to that platform's typical folder structure: WordPress disallows /wp-admin/, /wp-includes/, and search result URLs while allowing uploaded media; Next.js disallows /api/ and /_next/ (its internal build assets) while allowing everything else; Shopify disallows cart, checkout, and account paths that shouldn't be crawled or indexed.

These templates use a placeholder sitemap URL of https://example.com/sitemap.xml, which you need to replace with your actual domain and sitemap path before using the file — the tool doesn't automatically detect or substitute your real domain. Since the template loads into an editable text box, you can also add, remove, or adjust any line directly rather than being limited to the platform's default rule set.