SEO · Free · No sign-up

robots.txt Generator for WordPress, WooCommerce & AI Crawlers

Pick your platform and what to block, and get a robots.txt file to copy or download — with separate controls for AI crawlers: the ones that train models and the ones that fetch answers from your site.

Platform
AI training crawlers

They collect content to train models. Blocking them doesn’t affect your Google rankings.

AI search & answer crawlers

They read your pages to cite them in ChatGPT, Perplexity and Claude answers. Blocking them can remove you from those answers.

robots.txt

Everything runs in your browser. Nothing you enter is sent anywhere.

What is robots.txt?

robots.txt is a small text file at the root of your site (yoursite.com/robots.txt) that tells search engine and AI crawlers which parts they may visit. Every well-behaved crawler reads it before it starts.

Keep in mind that robots.txt blocks crawling, not indexing. A page blocked in robots.txt can still appear in Google if other pages link to it. To keep a page out of Google, use noindex in your SEO plugin and leave the page crawlable so Google can see the noindex.

The presets

  • WordPress — blocks /wp-admin/ and allows admin-ajax.php, which some plugins need.
  • WooCommerce — the WordPress rules plus cart, checkout and account pages, and the add-to-cart and sorting URLs that create thousands of duplicates.
  • Internal search — your site’s search result pages add nothing in Google and waste crawl time.

AI crawlers: what to block and what to allow

There are two kinds, and it pays to tell them apart:

  1. Training crawlers such as GPTBot, ClaudeBot, Google-Extended and CCBot collect content to train models. Blocking them is your call and doesn’t affect your Google rankings.
  2. Search and answer crawlers such as OAI-SearchBot, PerplexityBot and Claude-SearchBot read your page when someone asks a question, so they can cite it. Block them and your site may stop appearing in ChatGPT, Perplexity and Claude answers.

The Content-Signal line is a newer way to state your preferences: allowed for search, allowed as input for AI answers, not allowed for training. Not every crawler honours it yet, but it does no harm.

If your site runs on WordPress and you want AI tools to read your content cleanly, see the free Rawi MD plugin and the llms.txt generator.

Frequently asked questions

Where do I put robots.txt?

At the root of your site (usually public_html), so it opens at yoursite.com/robots.txt. With Yoast you can also edit it under Yoast → Tools → File editor.

Does blocking GPTBot hurt SEO?

No. GPTBot crawls for OpenAI model training and has nothing to do with Google rankings. OAI-SearchBot powers ChatGPT search results, so blocking that one can keep you out of them.

How do I hide a page from Google?

Use noindex from your SEO plugin, not robots.txt. If you block the page in robots.txt, Google can’t read the noindex and the page may stay in results.

What happens if my site has no robots.txt?

Crawlers assume everything is allowed. That isn’t a big problem for a small site, but the file helps steer crawlers away from useless pages and points them to your sitemap.

النسخة العربي من الأداة