Robots.txt Generator

Create a robots.txt file without memorising its syntax. Start from allow-all, block-all or custom rules, add groups for specific crawlers such as Googlebot or GPTBot, list the paths to allow or block and add your sitemap URLs. The tool validates every path, previews the file and lets you test whether a URL would be crawled.

How to use the Robots.txt Generator

1. Choose allow all, block all or custom rules.
2. For custom rules, set a user-agent and list paths to disallow or allow.
3. Add more crawler groups or a crawl-delay if needed.
4. Enter your sitemap URLs.
5. Test a few URLs, then download robots.txt and upload it to your site root.

Features

• Per-user-agent groups with suggestions for common crawlers, including AI bots
• Allow and disallow paths with * and $ wildcard support
• Path validation with clear error messages
• Crawl-delay and multiple sitemap lines
• URL tester using Google's longest-match rule precedence
• Copy or download the finished file

Benefits

• Avoid syntax mistakes that accidentally block your whole site
• Control which crawlers, including AI training bots, may access your content
• Help search engines find your sitemap

Frequently asked questions

Does robots.txt keep pages out of Google?

Not reliably. It stops crawling, but a blocked URL can still be indexed if other sites link to it. Use a noindex meta tag on a crawlable page to keep it out of results.

Does Google respect Crawl-delay?

No. Googlebot ignores Crawl-delay. Bing and Yandex honour it.

Where do I put robots.txt?

At the root of your domain, for example https://example.com/robots.txt. Each subdomain needs its own file.

Is robots.txt a security measure?

No. It is a public request that well-behaved crawlers follow. Anyone can read it, so never use it to hide private pages; protect them with authentication.

Related tools

• Slug Generator — Turn titles into clean, SEO-friendly URL slugs with accent conversion, stop-word removal, length limits and batch mode.
• Reading Time Calculator — Estimate how long an article takes to read or say aloud at slow, average, fast or custom speeds, and copy an "X min read" label.
• Meta Tag Generator — Create title, description, robots, canonical, Open Graph and X card meta tags with length meters and live search and social previews.
• Keyword Density Checker — Find the most frequent one-, two- and three-word phrases in your content, with density percentages and target-keyword highlighting.
• Percentage Calculator — Work out X% of Y, what percent one number is of another, percentage change, or increase and decrease a value by a percent.
• Loan EMI Calculator — Calculate the monthly EMI for any loan, plus total interest, total payment and a year-by-year repayment breakdown.