About this robots.txt generator
A robots.txt file sits at the root of your site and tells crawlers which paths they may fetch. It is made of groups: one or more User-agent lines naming the crawlers, followed by Allow and Disallow rules for them. It also lists your sitemaps. A mistake can hide a whole site from search results, so this tool both builds files and checks them. Generate mode writes a clean file from simple fields, turns relative sitemap paths into full URLs, and warns about rules that look dangerous. Validate mode reads an existing file, lists its crawler groups, and flags broken lines.
How to create a robots.txt file
- Choose Generate or Validate. Pick Generate robots.txt to build a file, or Validate robots.txt to check one you already have. Press an example to see both in action.
- Fill in the rules. Enter your Site URL, the Main user-agent (usually *), and the paths to allow and disallow, one per line. Add Sitemap URLs, or paste a file to validate.
- Add optional lines. Use Crawl-delay, Host, Clean-param rules, and Extra user-agent blocks only if you need them.
- Press the button. Copy or download the file and upload it to the root of your site as robots.txt. Read the Warnings and notes first.
Frequently asked questions
Where do I put the robots.txt file?
At the root of the host it applies to, so it is reachable at https://example.com/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own file.
Does Disallow remove a page from Google?
No. It stops crawlers from fetching the page, but a blocked URL can still appear in results if other sites link to it. To keep a page out of search, use a noindex tag and let it be crawled.
Why does the tool warn about blocked CSS and JavaScript?
Search engines render pages like a browser. If scripts, styles, or images are disallowed they cannot see the page as visitors do, which can hurt how it is understood and ranked.
Do all crawlers follow Crawl-delay?
No. Bing and some other crawlers follow it, but Google ignores it, and Google’s crawl rate is managed in Search Console instead.
What does an empty Disallow line mean?
A Disallow line with nothing after it means nothing is blocked, so the group allows everything. It is the standard way to say that all crawling is permitted.