Super Boring Tools

Search tools

Find a tool by name or what it does.

robots.txt Generator

Build a valid robots.txt with grouped crawler rules, crawl-delay, and a sitemap line.

Quick presets

Load a starting point, then refine the rules below. Everything runs in your browser. Nothing is uploaded.

User-agent groups

User-agent
seconds (optional)

Sitemap

robots.txt

What this file does

  • All crawlers may not crawl /admin, /private.
  • A sitemap location is shared with every crawler.
  • robots.txt is a request, not a lock. Well-behaved crawlers obey it, but it does not stop anyone from opening a URL, so protect private pages with a login.
  • Paths are case-sensitive. /Admin and /admin are treated as different.

How to create a robots.txt file online

  1. Pick a preset

    Start from allow all, block all, or block AI crawlers, or build your own rules from scratch.

  2. Add your rules

    For each user-agent group, add allow and disallow paths and an optional crawl-delay.

  3. Add your sitemap

    Enter the full address of your sitemap so crawlers can find your list of pages.

  4. Copy or download

    Read the plain-English summary to confirm the rules, then copy the file or save it as robots.txt.

Why use this tool

Grouped user-agent blocks

Add a separate block for every crawler, each with its own allow and disallow rules and an optional crawl-delay.

Ready-made presets

One click sets up allow all, block all, or a block list for common AI crawlers, then you refine from there.

Plain-English summary

A short explanation under the file spells out exactly which paths each crawler can and cannot reach.

Path and sitemap checks

Missing leading slashes are fixed for you, and warnings flag spaces in paths, invalid crawl-delay values, and sitemap URLs that are not absolute.

Copy or download instantly

The file rebuilds as you type, so copy it to the clipboard or save it as robots.txt with one click.

Runs entirely in your browser

Everything happens on your device; nothing is uploaded.

About this tool

A robots.txt file sits at the root of your domain and tells search engines and other automated crawlers which parts of your site they may request. This generator turns a few simple inputs, one or more user-agent groups with allow and disallow paths, an optional crawl-delay, and your sitemap address, into a correctly formatted file with one directive per line and a trailing sitemap reference.

Reach for it when launching a site, hiding a staging or admin area from search results, slowing down an aggressive crawler, or opting out of the large language model bots that have appeared in the last few years. The presets cover the three most common cases, allow everything, block everything, and block a curated list of AI crawlers, and you can then add, remove, and edit rules by hand. Every path is normalized to start with a slash, and a plain-English summary restates the rules so you can confirm the file does what you intended before you publish it.

Remember that robots.txt is advisory. Reputable crawlers honor it, but it does not stop anyone from opening a URL, and path matching is case-sensitive. For pages that must stay private, use a login instead, and to control indexing of a single page, pair this with the meta tag generator. When you are collecting the sitemap and page addresses by hand, the URL parser helps you check each one.

Frequently asked questions

What is a robots.txt file?
It is a plain text file placed at the root of your domain, at /robots.txt, that tells crawlers which parts of your site they may request. It groups rules under one or more user-agent lines and can also point to your sitemap.
Does robots.txt actually block crawlers?
It is advisory, not a security control. Reputable search engines and bots obey it, but it does not stop anyone from opening a URL directly. For pages that must stay private, protect them with a password or login instead.
How do I block AI crawlers?
Use the block AI crawlers preset. It adds a separate block for common model bots, each set to disallow the whole site, while still allowing other crawlers. You can add or remove agents to match the list you want to opt out of.
Where do I put the file and does path case matter?
Upload it to the root of your domain so it is reachable at https://yoursite.com/robots.txt. Paths are case-sensitive, so /Admin and /admin are treated as different, and every path should start with a slash.
Can I add a crawl-delay and multiple sitemaps?
Yes. Each user-agent group can set an optional crawl-delay in seconds, and you can list more than one sitemap. Sitemap lines go at the end of the file and must use absolute URLs.
Is my data uploaded anywhere?
No. Everything runs in your browser; nothing is sent to a server. The rules and URLs you type never leave your device and are not stored or logged.

Related tools