Robots.txt Generator
Set up user-agent groups with Allow and Disallow rules, add your sitemap, and copy or download the file. A built-in tester tells you whether a given path is allowed for a given crawler.
Start from a preset
The AI preset blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent. Companies add and rename crawlers, so check their own pages now and then.
Group 1
* means every crawler. Separate several names with commas; each gets its own User-agent line in the same group.
Rules
Google ignores Crawl-delay. Bing and some other crawlers read it.
Group 2
* means every crawler. Separate several names with commas; each gets its own User-agent line in the same group.
Rules
Google ignores Crawl-delay. Bing and some other crawlers read it.
One full URL per line. Each becomes a Sitemap line at the end of the file.
robots.txt
User-agent: * Disallow: /admin/ Allow: /admin/help/ Disallow: /cart/ Disallow: /checkout/ Disallow: /*.pdf$ User-agent: Bingbot Disallow: /search Crawl-delay: 5 Sitemap: https://www.example.com/sitemap.xml
Reminder: Googlebot skips Crawl-delay. Set crawl limits for Google in Search Console instead.
Test a path
Blocked
Using the User-agent: * group. Deciding rule: Disallow: /admin/ (line 2).
Follows Google's rules: the group naming the crawler beats *, the longest matching rule wins, and Allow wins a tie. Paths are case-sensitive.
Runs in your browser
The file and the tests are made in this tab. Nothing is uploaded.
Google's matching rules
Longest rule wins, Allow wins ties, * and $ work as Google reads them.
Ready to upload
Copy the text or download a robots.txt file for your site's root.
A small file that crawlers read first
Before a search engine crawls your pages, it asks for /robots.txt at the root of your domain. The file lists groups: each starts with one or more User-agent lines naming the crawlers it's for, followed by Allow and Disallow lines with path patterns. A crawler picks the group meant for it and skips the paths that group disallows.
Keep in mind that robots.txt is a request, not a lock. Googlebot, Bingbot and other well-behaved crawlers follow it, but nothing stops a badly behaved bot from ignoring it, and anyone can open the file and read the paths you list. If a folder holds private data, protect it with a login or server rules. Also, blocking a page doesn't guarantee it stays out of search results: Google can still index a blocked URL that other sites link to, just without its content. Use a noindex tag for that, and leave the page crawlable so the tag can be seen.
This generator builds the file as you edit and keeps the tester pointed at the exact text in the output box, so what you test is what you'll upload. It all happens in your browser.
How to use it
- 1Pick a preset, or start from the sample and edit it.
- 2For each group, enter the user-agent (* for every crawler) and add Allow or Disallow rules.
- 3Add a Crawl-delay if a crawler such as Bingbot hits your server too hard, and list your sitemap URLs.
- 4Type a path and a crawler name into the tester to check the result.
- 5Copy the text or download robots.txt and upload it to the root of your site, so it opens at yourdomain.com/robots.txt.
The presets
- Allow everything: one group for * with an empty Disallow line, which lets every crawler fetch every page. Handy when you only want to list a sitemap.
- Keep everything out: Disallow: / for every crawler. Use it on staging sites or a site that isn't ready yet, together with a password if the content is private.
- Keep private paths out: blocks /admin/, /login/, /account/, /cart/, /checkout/, /search and URLs with a ?sort= parameter. Edit the list to match the folders your site actually uses.
- Opt out of AI training: adds a group that disallows everything for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl) and meta-externalagent (Meta). These are tokens the companies publish for site owners. Google-Extended and Applebot-Extended don't affect normal search crawling. Other bots exist, and names change, so check each company's crawler page from time to time.
How the tester decides
The tester follows the rules Google documents. First it picks the group: a group that names the crawler (compared without case, and ignoring a version like /2.1) wins over the * group, and several groups for the same crawler are merged. If no group applies, everything is allowed.
Then it looks at every Allow and Disallow rule in that group that matches the start of the path. The longest rule wins, and when an Allow and a Disallow are the same length, Allow wins. A * in a rule matches any run of characters, and a $ at the end means the path has to end there, so Disallow: /*.pdf$ blocks /files/menu.pdf but not /files/menu.pdf?v=2.
Paths are case-sensitive, so /Admin/ and /admin/ are different. If you paste a full URL, the tester uses only its path and query string.
Crawl-delay and sitemaps
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and some smaller crawlers respect it. Google ignores the line completely and adjusts its crawl rate on its own, so there's no point setting it for Googlebot.
Sitemap lines aren't tied to any group, so the generator writes them at the end of the file. Use full addresses, including https://. You can list more than one, for example a separate sitemap for images or news.
Mistakes worth avoiding
- Leaving Disallow: / in place after a site moves from staging to live. It happens more than you'd think, and traffic drops fast.
- Blocking CSS or JavaScript folders. Google needs them to render your pages properly.
- Listing secret folders. The file is public, so you're giving out a map.
- Putting robots.txt in a subfolder. Crawlers only look at the root of each host, and each subdomain needs its own file.
Related tools
What Our Users Say
Real feedback from people using WeblixTools, checked by our team before it appears.
5.0 out of 5 · 1 review
very nice
Frequently Asked Questions
Where does robots.txt go?
In the root folder of your site, so it opens at https://yourdomain.com/robots.txt. Crawlers don't look for it anywhere else. A subdomain such as shop.yourdomain.com needs its own copy.
Does robots.txt stop people or bots from seeing a page?
No. It's a request that well-behaved crawlers follow, not access control. Anyone can still open the page in a browser, and bad bots can ignore the file. Use a password or server rules for anything private.
Will Disallow remove a page from Google?
Not reliably. Google won't crawl it, but it can still index the URL if other pages link to it. To keep a page out of results, allow crawling and add a noindex robots meta tag instead.
Does Google follow Crawl-delay?
No, Googlebot ignores it. Bing and some other crawlers do read it, which is why the field is there. Leave it empty unless a crawler is putting real load on your server.
Can I block AI crawlers with robots.txt?
You can ask the ones that publish a user-agent token for it. The AI preset covers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. It only works for crawlers that choose to honour robots.txt, and it doesn't remove anything already collected.
What happens if Allow and Disallow both match a path?
Google uses the longest matching rule. With Disallow: /admin/ and Allow: /admin/help/, the path /admin/help/faq is allowed because the Allow rule is longer. If both rules are the same length, Allow wins.
Is the tester the same as Google's own?
It follows the same documented rules for picking a group and matching paths, so it's a reliable check for normal files. It doesn't decode percent-encoded characters or fetch your live file, so for edge cases confirm with the robots.txt report in Google Search Console.