Robots.txt Generator

Build a robots.txt file in the browser

Robots.txt tells crawlers which paths they may fetch. Add a user-agent, an allow path, a disallow path, an optional crawl delay, and a sitemap URL. The file updates as you edit. Presets cover a normal public site, allow everything, block everything, or block common AI crawlers.

How the rules are grouped

Rows that share a user-agent are merged into one block. An empty allow or disallow field is left out. If a row has neither path, the file writes an empty Disallow: line, which means that agent is not blocked. A crawl delay above zero is added to every group. Put the finished file at the root of the site as /robots.txt.

Related SEO tools

Check how an AI crawler is treated with the GPT Bot Crawl Tester, pull URLs from an existing file with the Sitemap URL Extractor, and write page tags with the Meta Tag Generator.