Free tool
Free Robots.txt Generator
Set user-agent, allow and disallow paths and your sitemap URL. Copy a ready-to-upload robots.txt file.
Inputs
Live GeneratorGenerated output
A robots.txt file tells crawlers which parts of your site they may visit. It’s a small text file with a big blast radius: one wrong line can hide a whole site from search. Here’s how to create one, put it live and check it.
How to create and upload robots.txt
- Set the rules above and click Generate robots.txt.
- Copy the output into a plain text editor and save it as
robots.txtin UTF-8. Word processors can add characters that break the file. - Upload it to the root of your site, so it loads at
https://yourdomain.com/robots.txt. A file in a subfolder is ignored. - Open that URL in a private browser window to confirm it’s public, then check the robots.txt report in Search Console.
Google covers each step in How to write and submit a robots.txt file.
One group per crawler
The generator writes one group: a User-agent line followed by its rules. To set different rules for different crawlers, generate each group and stack them in one file with a blank line between them.
User-agent: *
Disallow: /admin/
Disallow: /cart
User-agent: GPTBot
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xmlRules that trip people up
- Paths are case-sensitive.
Disallow: /fishblocks/fish.htmlbut not/Fish.asp. - Paths are prefixes.
Disallow: /blogalso blocks/blog-archive. Add the trailing slash to block only the folder. - The most specific rule wins. Google applies the longest matching path, so
Allow: /admin/help/beatsDisallow: /admin/. Disallow: /blocks the whole site. Fine on a staging site. A disaster if it ships to production.- Don’t block CSS or JavaScript Google needs to understand the page.
The full syntax is in Google’s robots.txt specification, which also confirms Google ignores Crawl-delay.
Robots.txt is not a way to hide pages
| Method | Stops crawling | Keeps it out of search | Use it for |
|---|---|---|---|
| robots.txt Disallow | Yes | No | Admin, cart and search-result pages |
| noindex tag | No | Yes | Pages that must not appear in results |
| Password protection | Yes | Yes | Private or customer-only content |
Google’s introduction to robots.txt is explicit: a disallowed page can still be indexed if it’s linked from elsewhere.
Blocking AI crawlers
Each AI company publishes its own user agents, and they separate training from search:
# Gemini training. Does not affect Google Search.
User-agent: Google-Extended
Disallow: /
# OpenAI model training
User-agent: GPTBot
Disallow: /
# Anthropic model training
User-agent: ClaudeBot
Disallow: /Sources: Google’s list of common crawlers, OpenAI’s crawler overview and Anthropic’s crawler help article. If you want to keep appearing in ChatGPT search, leave OAI-SearchBot allowed.
Shopify stores
Shopify serves a default robots.txt for every store. You change it by editing a robots.txt.liquid theme template, not by uploading this file. Shopify’s guide to editing robots.txt warns that a wrong edit can cost you all your traffic.
Next, point crawlers at the pages you do want found. The XML sitemap generator builds the file your Sitemap: line refers to.
FAQ
Frequently asked questions
Straight answers about using this tool and its output.
How do I create a robots.txt file?
Set your rules above and click Generate. Copy the output into a plain text file named robots.txt, saved as UTF-8. Upload it to the root of your site so it loads at https://yourdomain.com/robots.txt. Each host can have only one robots.txt file.
Can robots.txt keep a page out of Google?
No. Robots.txt controls crawling, not indexing. Google says a disallowed page can still be indexed if other sites link to it. To keep a page out of search results, use a noindex tag or put it behind a password, and don’t block it in robots.txt, or Google won’t see the noindex.
Should I add my sitemap URL to robots.txt?
Yes. A line like Sitemap: https://yourdomain.com/sitemap.xml is one of the ways Google accepts to find your sitemap, and other crawlers read it too. The generator adds it at the end of the file.
Does Google follow Crawl-delay?
No. Google’s robots.txt documentation lists crawl-delay as unsupported. Some other crawlers, such as Anthropic’s ClaudeBot, document support for it. Leave the field empty unless a specific crawler is putting load on your server.
How do I block AI training crawlers without leaving Google Search?
Add a group for each crawler: Google-Extended for Gemini training, GPTBot for OpenAI and ClaudeBot for Anthropic, each with Disallow: /. Google states that Google-Extended does not affect inclusion or ranking in Google Search. OpenAI uses a separate crawler, OAI-SearchBot, for ChatGPT search results.
I use Shopify. Can I upload my own robots.txt?
Shopify already serves a default robots.txt. To change it you edit a robots.txt.liquid theme template rather than uploading a file. Shopify warns that a wrong edit can cost you all your search traffic, so keep changes small.
Related tools
View all toolsLet ZynfoAI take the first reply.
Answers on your website and WhatsApp from your own content, with your team one click away. Free for 7 days, then $99/month.