Quick answer
To create a robots.txt file, choose the default rule for all robots (allow everything or block the whole site), list folders or pages to block — use presets for WordPress admin, search results, cart and checkout, login and URL parameters — add any paths to always allow, an optional crawl-delay and your sitemap URLs. Then set individual rules for search engines and AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. Copy or download the file and upload it to your site's root.
- Default rule:
- Allow everything or block the whole site
- Block and allow:
- Any folders or pages, with presets
- Sitemaps:
- One or more sitemap URLs
- Search engines:
- Googlebot, Bingbot, Googlebot-Image, Yandex, Baidu, DuckDuckBot
- AI crawlers:
- GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more
- Extras:
- Optional crawl-delay
- Output:
- Copy or download robots.txt
- Price:
- Free
Every website should have a robots.txt file. It sits at the root of your domain — https://yourdomain.com/robots.txt — and is the first file well-behaved crawlers read. It tells search engine bots and other crawlers which parts of your site they may visit and where to find your sitemap. Used well, it keeps crawlers away from admin areas, internal search results and endless filter URLs, so they spend their time on the pages that matter. Used badly, a single line can hide your entire site from Google.
Robots.txt has also become the main way to express your choice about AI crawlers. Companies building AI models and AI search engines send their own bots — OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, Common Crawl's CCBot and others — and many respect robots.txt rules. Google and Apple offer special tokens (Google-Extended, Applebot-Extended) that let you opt out of AI training without affecting normal search.
Our robots.txt generator lets you build the file with clear choices instead of syntax. Set a default, add blocked paths with one-click presets, add sitemap URLs and decide for each search engine and AI crawler whether to allow it, block it or follow the default. The file updates live as you choose, ready to copy or download.
How to create a robots.txt file – step by step
- 1 Choose the defaultAllow everything for normal websites, or Block the whole site for staging and private sites.
- 2 Block folders and pagesType one path per line, or click presets such as WordPress admin, Search results or Cart & checkout.
- 3 Add allow rules and sitemapsAdd paths that must stay crawlable (for example admin-ajax.php) and your sitemap URLs.
- 4 Set search engine and AI rulesFor each bot choose allow, block or same as default. Use "allow all" or "block all" for AI crawlers in one click.
- 5 Copy, upload and testDownload robots.txt, upload it to your site's root folder and check it in Google Search Console's robots.txt report.
Why use this robots.txt generator?
No syntax errors
Valid directives generated from simple choices.
Handy presets
WordPress admin, search, cart, login and URL parameters in one click.
AI crawler control
Allow or block GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more.
Search engine rules
Individual settings for Googlebot, Bingbot, Yandex, Baidu and others.
Sitemap support
Add one or several sitemap URLs.
Free and instant
Live preview, copy or download.
Robots.txt syntax in brief
User-agent:names the crawler a group of rules applies to;*means all crawlers.Disallow:blocks a path.Disallow: /blocks everything; an empty Disallow allows everything.Allow:re-opens a path inside a blocked folder, for exampleAllow: /wp-admin/admin-ajax.php.Sitemap:gives the full URL of a sitemap; you can list several.*matches any characters and$marks the end of a URL, soDisallow: /*.pdf$blocks PDF files.Crawl-delay:asks crawlers to wait between requests. Bing and Yandex respect it; Google ignores it.
Blocking vs. noindex — an important difference
Robots.txt controls crawling, not indexing. If you block a page in robots.txt, Google cannot read it — but it can still index the URL (without content) if other pages link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header instead. Never block a page in robots.txt and expect its noindex tag to work: Google cannot see the tag if it cannot crawl the page. Also never block CSS and JavaScript files your pages need — Google renders pages like a browser and needs them.
Recommended robots.txt for WordPress
For most WordPress sites a minimal file is best: allow everything, block /wp-admin/ but allow /wp-admin/admin-ajax.php, optionally block internal search results (/?s=) and list your sitemap (usually /sitemap_index.xml with Rank Math or Yoast, or /wp-sitemap.xml). Do not block /wp-content/ or /wp-includes/, because they contain images, CSS and scripts. WooCommerce stores often block /cart/, /checkout/ and /my-account/, which have no search value.
Should you block AI crawlers?
It depends on your goals. Blocking AI training bots (such as GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended) signals that you do not want your content used to train AI models. On the other hand, AI search and assistant bots — such as OAI-SearchBot, ChatGPT-User, Claude-SearchBot and PerplexityBot — fetch pages to answer users' questions and can send you visitors and citations. Many publishers block training bots while allowing search bots; others allow everything to maximise visibility in AI answers. Blocking Google-Extended does not affect your Google Search rankings. Remember that robots.txt is voluntary: reputable companies follow it, but it is not a technical barrier.
Common robots.txt mistakes
- Leaving "Disallow: /" after launch — staging settings copied to the live site hide everything from Google.
- Blocking CSS/JS — prevents Google from rendering pages properly.
- Using robots.txt to hide private content — the file is public and lists the paths you want hidden. Use passwords instead.
- Wrong location — it must be at the root: example.com/robots.txt, not example.com/blog/robots.txt.
- Case and trailing slashes — paths are case-sensitive; /Admin/ and /admin/ are different.
- One file for all subdomains — each subdomain (shop.example.com) needs its own robots.txt.
How to upload and test robots.txt
Upload the file to your website's root folder via your hosting file manager or FTP and open https://yourdomain.com/robots.txt to confirm it loads as plain text. In WordPress, SEO plugins such as Rank Math and Yoast include a robots.txt editor, which is easier than uploading a file. Check Google Search Console's robots.txt report (Settings → robots.txt) to see the version Google fetched and any errors. Our SEO Report confirms whether a robots.txt and sitemap are found, and the Sitemap Generator can create a sitemap to list in it.
When to block the whole site
Block everything (Disallow: /) only for development, staging and test sites that should never appear in search results — and add password protection too, because some crawlers ignore robots.txt. Remember to change it when the site goes live; this is one of the most common causes of a new site not appearing in Google.
Troubleshooting
- Google still shows a blocked page. Blocking stops crawling, not indexing. Use noindex and allow crawling.
- Search Console says "Blocked by robots.txt". Check whether that page should be blocked; if not, remove or narrow the rule.
- Changes do not apply. Google caches robots.txt for up to 24 hours. Request a recrawl in Search Console.
- The file downloads instead of displaying. Make sure it is named robots.txt and served as text/plain.
Frequently asked questions
What is a robots.txt file?
A text file at your site's root that tells crawlers which parts of the site they may visit.
Do I need a robots.txt file?
It is recommended. Without one, crawlers assume everything is allowed, which is fine for many small sites.
Where do I put robots.txt?
In the root folder, so it opens at yourdomain.com/robots.txt.
Can robots.txt remove a page from Google?
No. Use a noindex tag and allow crawling. Robots.txt only blocks crawling.
How do I block AI crawlers?
Set GPTBot, ClaudeBot, CCBot, Google-Extended and others to block in the AI crawler section.
Will blocking Google-Extended hurt my rankings?
No. It only affects use of your content for Google's AI models, not Google Search.
Does Google respect crawl-delay?
No. Bing and Yandex do; Google manages crawl rate automatically.
Should I add my sitemap to robots.txt?
Yes. It helps all search engines find your sitemap.
What is the best robots.txt for WordPress?
Allow everything, block /wp-admin/ while allowing admin-ajax.php, and list your sitemap.
Is robots.txt a security feature?
No. It is public and voluntary. Protect private content with passwords.
Is the generator free?
Yes. No sign-up required.
Does each subdomain need its own robots.txt?
Yes. shop.example.com and blog.example.com each need their own file at their own root.
Last updated November 22, 2026 · Written and tested by the SmallEasyTools team.