Robots.txt Generator
Build a complete robots.txt file with multiple user-agent blocks, AI bot rules, and sitemap reference.
Configure user-agent blocks with allow and disallow paths, optional crawl-delay, and a sitemap reference. The syntax-highlighted preview updates live as you build.
Quick presets
User-agent: *Allow: /Disallow: /admin/Disallow: /private/Disallow: /tmp/ Sitemap: https://example.com/sitemap.xml
Test a URL path
See which rule applies for any path + bot.
Place this file at yoursite.com/robots.txt. Robots.txt is a crawl hint — well-behaved bots respect it but it is not a security barrier, and a blocked URL can still be indexed if other pages link to it (use noindex on a crawlable page to keep it out of results).
Saved configurations
Use the "Save configuration" button above to save your current setup here.
What robots.txt does and why it matters
A robots.txt file sits at the root of your domain and tells well-behaved crawlers which pages they are and are not allowed to crawl. It is a widely respected convention that gives site owners basic control over crawl behavior without requiring server-level access. While it is not a security mechanism — determined bots can ignore it — it is effective for legitimate indexers like Googlebot, Bingbot, and most trusted crawlers.
A properly structured file helps protect admin panels and staging paths from appearing in search results, reduces crawl budget waste on low-value pages, and lets you send separate instructions to different bots. This generator creates the full file with multiple user-agent blocks, AI-specific rules, crawl-delay hints, and a sitemap reference — all with a syntax-highlighted live preview before you download.
Blocking AI training crawlers
A growing category of robots.txt rules targets AI crawlers such as GPTBot, CCBot, ClaudeBot, and ChatGPT-User. Unlike traditional search crawlers, many of these bots collect data for model training or AI answers rather than search indexing. Many site owners now choose to disallow them by default. The "Block AI bots" preset gives you a quick starting point for that setup — it keeps normal search crawlers unrestricted while blocking known AI crawlers.
You can freely combine rules by adding multiple blocks. A wildcard block can allow most crawlers through while specific blocks restrict individual bots differently. Each block supports its own paths and an optional crawl-delay directive. Note that a crawler obeys only the single most specific group that names it, so a named block does not inherit rules from the * block, and Google ignores crawl-delay. When done, download the file and place it at the root of your domain at yoursite.com/robots.txt.
Create a robots.txt file in minutes
This robots.txt generator builds a valid robots.txt file for your website without writing rules by hand. Add rules for one or many crawlers, block AI bots, add your sitemap, test any URL path, then copy the file or download it as robots.txt.
How to make a robots.txt file
- Enter your site URL. A
Sitemap:line pointing to/sitemap.xmlis added automatically, or you can enter a different sitemap URL. - Start from a preset — Allow all, Block all, Block AI bots or SEO default — or build your own.
- Add rule blocks. In each block choose the user-agents (Googlebot, Bingbot, GPTBot and others), then add Allow and Disallow paths and an optional crawl-delay.
- Use Test a URL path to check whether a page is allowed or blocked for a given bot.
- Download the file and upload it to the root of your domain, so it opens at
https://yourdomain.com/robots.txt.
Robots.txt example
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xmlThis file lets every crawler visit the site except the admin and private folders, blocks OpenAI GPTBot completely, and tells search engines where the sitemap is.
Robots.txt rules explained
- User-agent — the crawler the block applies to.
*means every bot without its own block. - Disallow — a path the bot should not crawl. An empty
Disallow:allows everything. - Allow — re-opens a path inside a disallowed folder. When rules conflict, the longest matching path wins.
- Crawl-delay — seconds between requests. Bing respects it; Google ignores it.
- Sitemap — the full URL of your XML sitemap.
Common robots.txt mistakes
- Blocking CSS and JavaScript. Google needs your scripts and styles to render pages. Blocking folders such as
/assets/,/static/or/_next/can make pages look empty to Google and hurt rankings. - Using Disallow to hide a page from search. Disallow stops crawling, not indexing. To keep a page out of Google, use a
noindexmeta tag and leave it crawlable. - Leaving “Disallow: /” from a staging site on the live site, which blocks the whole website.
- Wrong location. The file only works at the domain root, not in a sub-folder.
Should you block AI bots?
AI crawlers such as GPTBot, CCBot, Google-Extended and anthropic-ai collect content for AI models. Blocking them does not affect your Google rankings. Some assistants also send visitors by citing sources, so decide per bot. The Block AI bots preset blocks the nine most common AI crawlers while keeping search engines allowed.
Frequently asked questions
Do I need a robots.txt file?
It is not required, but it is recommended: it points crawlers to your sitemap and keeps them out of admin, search and duplicate pages.
Where do I upload robots.txt?
In the root folder of your website, next to your home page, so it is reachable at /robots.txt.
How do I check my robots.txt is working?
Open yourdomain.com/robots.txt in a browser, then use the robots.txt report in Google Search Console to see the version Google fetched.
Related Developer Tools
Browse all tools from the same category.
Domain Authority Checker
Check Moz authority, page authority, spam score, and backlink metrics
Open toolSEO Audit Tool
Audit a website URL for technical SEO, metadata, headings, broken internal links, hreflang, canonical and X-Robots-Tag checks, with report exports
Open toolShowing 1-2 of 48