How to use the AI Crawler robots.txt Generator
The generator runs in your browser and builds a block of AI robots.txt rules from the choices you make. Nothing is uploaded, and the result is plain text you can paste into your existing file.
- Start from a preset if one fits: allow all AI bots, block AI training only, or block all AI bots. Presets set a sensible default for every crawler in the list, and you can adjust them afterward.
- Review the list of AI crawlers. Each entry shows the bot name, the company that operates it and what it does, such as training, search indexing or fetching a page for a user.
- For each crawler, choose Allow or Block. If you want a middle path, such as keeping the crawler out of a private area, use the path fields to list the folders it should not enter.
- Optionally add your sitemap URL so the output includes a
Sitemap:line. - Click Generate and read the preview. Rules are grouped by company, and each group has a comment explaining what the bot does.
- Copy the output or download it as a text file.
- Add the rules to the
robots.txtfile at the root of your site. If you already have a file, merge the new groups into it rather than replacing it, and keep your existing rules for search engines. - Open
https://yourdomain.com/robots.txtin a browser to confirm the new rules are live, then test with the AI Bot User-Agent Checker or your own server logs.
Tip: Back up your current robots.txt before editing it. A single misplaced line, such as Disallow: / under User-agent: *, can block every crawler, including Google.
What robots.txt does, and what it does not
robots.txt is a plain text file at the root of your domain that tells automated visitors which parts of the site they may request. It follows the Robots Exclusion Protocol, which has existed since the 1990s and is now documented as an internet standard. Each rule group begins with one or more User-agent lines naming the crawler, followed by Allow and Disallow lines for paths.
The most important thing to understand is that robots.txt is a request, not a lock. A crawler that chooses to follow the protocol reads the file before fetching pages and respects your rules. A crawler that does not follow it can ignore the file entirely, and nothing in the protocol stops it. Reputable operators such as OpenAI, Anthropic, Google, Apple and Perplexity say their main crawlers honor robots.txt. Some less transparent crawlers have been widely reported to ignore it.
That means robots.txt is the right tool for expressing your policy to well-behaved bots, and the wrong tool for security. If content must be private, protect it with authentication. If you need to stop an abusive bot that ignores your rules, you need server-level or network-level controls such as rate limiting, firewall rules or a bot-management service.
Why AI crawlers need their own rules
Until recently, most sites only thought about robots.txt in relation to search engines. Now a growing list of crawlers collects web content for AI systems, and they do not all do the same thing. Lumping them together leads to bad decisions in either direction: you either block everything and disappear from AI answers, or allow everything and give up control over how your content is used.
The key idea is that AI crawlers fall into different purposes, and you may want a different policy for each.
Training crawlers
These collect content that may be used to train or improve foundation models. Examples include GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl, Meta-ExternalAgent from Meta, Bytespider from ByteDance and Amazonbot from Amazon. Blocking them is how most site owners opt out of having their content included in future training data.
Search and index crawlers
These build the index that an AI search product uses to find and link to pages. Examples include OAI-SearchBot for ChatGPT search, Claude-SearchBot for Claude and PerplexityBot for Perplexity. Allowing these is what makes your pages eligible to be surfaced and cited in AI answers. OpenAI, Anthropic and Perplexity state that these bots are separate from their training crawlers.
User-triggered fetchers
These visit a page live when a person asks an assistant about it. Examples include ChatGPT-User, Claude-User and Perplexity-User. They behave more like a browser acting on behalf of a person than like a crawler that wanders the web. Vendors differ on whether robots.txt applies to them. Anthropic says Claude-User respects robots.txt, while OpenAI and Perplexity have indicated that their user-initiated fetchers may not follow robots.txt rules in the same way.
Control tokens
Two names you will see are not crawlers at all. Google-Extended and Applebot-Extended are tokens that you reference in robots.txt to tell Google and Apple how they may use content that their normal crawlers (Googlebot and Applebot) already collected. They have no user-agent string of their own and never appear in your logs. Blocking Google-Extended opts your content out of use for Gemini model training and grounding, and blocking Applebot-Extended opts it out of Apple foundation model training.
Because of these differences, the generator shows you each bot’s purpose beside its toggle. Choose by purpose, not just by name.
The crawlers the generator covers
The list below describes the crawlers you will most often meet. Operators change their bots over time, so confirm details in each vendor’s own documentation before you rely on them. The free aeoora WordPress plugin maintains a database of more than 300 AI crawler signatures, and the generator’s list reflects the most common ones.
- GPTBot (OpenAI) crawls public content that may be used to train OpenAI’s models. Respects robots.txt.
- OAI-SearchBot (OpenAI) crawls sites so they can be surfaced and linked in ChatGPT search results. Respects robots.txt.
- ChatGPT-User (OpenAI) fetches pages when a ChatGPT or custom GPT user asks. OpenAI notes that robots.txt may not apply to these user-initiated requests.
- ClaudeBot (Anthropic) collects public web content that may be used to train Claude models. Respects robots.txt.
- Claude-SearchBot (Anthropic) indexes content to improve the relevance of Claude’s search results.
- Claude-User (Anthropic) fetches a page live when a Claude user’s question requires it.
- PerplexityBot (Perplexity) indexes sites so they can be surfaced and cited in Perplexity answers. Perplexity states it is not used for training.
- Perplexity-User (Perplexity) visits pages when a user asks a question and generally does not follow robots.txt.
- Google-Extended (Google) is a control token that governs use of Google-crawled content for Gemini training and grounding.
- Applebot-Extended (Apple) is a control token that opts content out of Apple foundation model training. Applebot itself keeps crawling for Siri and Spotlight.
- CCBot (Common Crawl) builds the open Common Crawl archive, a dataset widely used to train language models.
- Bytespider (ByteDance) is widely reported to gather LLM training data and to be frequently observed ignoring robots.txt.
- Meta-ExternalAgent (Meta) crawls to train Meta’s AI models and index content for its products.
- Meta-ExternalFetcher (Meta) fetches individual links on user request and may bypass robots.txt.
- Amazonbot (Amazon) crawls to improve Amazon products and may be used to train Amazon AI models.
- Others include MistralAI-User, DuckAssistBot, YouBot and cohere-ai. Each follows its own documented rules.
Allow or block: how to decide
There is no single correct policy. The right answer depends on what you want from AI systems and what you are protecting. Here is a way to think it through.
If your goal is to be found, cited and recommended in AI answers, you want the search and fetch crawlers to reach your public pages. For most businesses that sell something or want inbound leads, being invisible to AI assistants is a real cost, because those assistants are becoming a place where buyers ask questions before they ever visit a site.
If your content is your product, such as a publication, a paid course or a proprietary dataset, you may reasonably decide that you do not want it used to train models without a license. That argues for blocking the training crawlers while leaving search and fetch bots alone.
If you are unsure, the middle path is common: allow search and user-triggered bots, block training-only bots, and revisit the decision every few months as vendors update their practices.
Three common policies work well as starting points:
- Open. Allow every AI crawler. Best for businesses that want maximum visibility and do not mind training use.
- Search yes, training no. Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user-triggered fetchers, and block GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider and Amazonbot. A balanced default for many publishers and brands.
- Closed. Block all AI crawlers. Best for content you license or sell. Expect to be absent from answers that rely on live retrieval, though the model may still reflect older content it already learned.
How AI engines treat your blocks
Blocking a crawler affects future access. It does not remove content that a model already learned, and it does not retroactively delete data a company already collected. Vendors generally say they stop collecting once they see the new rule, usually within a few days to a few weeks, depending on how often they refresh your robots.txt.
Blocking also does not always remove you from AI answers. An assistant may still describe your business using information from other sources: your social profiles, review sites, directories, news articles and third-party pages that mention you. If you block your own site, you remove the best source of accurate information and leave the field to those other sources.
A final nuance concerns Google. Google-Extended controls use of your content for Gemini training and grounding in certain products, but Google states that it does not affect inclusion in Google Search. AI features inside Search, such as AI Overviews, draw on content that Googlebot crawls, and Google documents separate controls for how snippets are shown. If you need to limit use there, read Google’s current documentation, because the rules differ from the Google-Extended token and change over time.
Best practices
Name each crawler explicitly
Use the exact token each operator publishes, such as GPTBot or ClaudeBot. Do not rely on wildcards to cover AI bots, because User-agent: * is usually written for search engines and many AI rules need to differ. A crawler follows the most specific group that matches its name, so named groups override the wildcard group.
Keep search-engine rules separate and safe
Do not touch the groups for Googlebot, Bingbot and other search crawlers unless you mean to. Generate AI rules as additional groups and place them in the same file. Never add a blanket Disallow: / to the wildcard group as a quick way to block AI.
Prefer path-level rules over all-or-nothing when it helps
You can allow a crawler into your public articles while keeping it out of account pages, internal search results and staging areas. Rules such as Disallow: /account/ and Disallow: /search reduce noise without hiding the content you want cited.
Group by operator and comment generously
Several operators run more than one crawler. Keeping their rules together, with a short comment explaining the purpose, makes later edits safer. A future teammate should be able to tell why GPTBot is blocked and OAI-SearchBot is allowed.
Add your sitemap
A Sitemap: line helps compliant crawlers discover the pages you want them to see. Use an absolute URL and keep it current.
Verify with logs, not assumptions
After you publish changes, watch your server logs or analytics for the bots you changed. A blocked crawler that stops appearing confirms the rule is working. A crawler that keeps hitting disallowed paths is ignoring you, and you may need to block it at the firewall or CDN. The AI Bot User-Agent Checker helps identify what a user-agent string in your logs actually is.
Remember that user-agent strings can be faked
Anyone can send a request that claims to be GPTBot. For decisions that matter, verify the visitor against the operator’s published IP ranges or reverse DNS, rather than trusting the label. Several vendors publish IP lists for this purpose.
Review on a schedule
New AI crawlers appear regularly, and existing ones change names or split into separate bots for training and search. Review your rules every few months. The generator’s list is updated as the landscape shifts, so regenerating periodically keeps you current.
Examples
These examples show common policies in valid robots.txt syntax. Paste only the groups you need, keep your existing search-engine rules and replace the sitemap URL with your own.
Allow search and user fetchers, block training
This policy keeps your pages eligible for AI search results while opting out of model training. It is the most common choice for publishers and brands.
# AI training crawlers: blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
# AI search and user-triggered fetchers: allowed
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
Block all AI crawlers
Use this when you license or sell your content and do not want any AI system to use it. It does not affect Googlebot or Bingbot.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
User-agent: meta-externalfetcher
User-agent: Bytespider
User-agent: Amazonbot
Disallow: /
Several User-agent lines in a row share the rule that follows them, which keeps the file compact.
Allow AI bots but protect private areas
This policy welcomes AI crawlers on public pages and keeps them out of areas that add no value to answers.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /cart/
Disallow: /checkout/
Disallow: /search
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
The final Allow line is a common WordPress exception that keeps AJAX features working for crawlers that render pages.
Testing your robots.txt
Mistakes in robots.txt are easy to make and easy to miss, so test before and after you deploy.
- Load the file in a browser. It should appear as plain text at the domain root, with no HTML wrapper and no login prompt.
- Check the status code. A 200 response is what you want. If robots.txt returns a server error, many crawlers will pause crawling the whole site, and a 404 is treated as “everything allowed.”
- Look for syntax slips. Directive names are case-insensitive but the paths you list are case-sensitive. A missing colon, a stray space or a rule placed before any
User-agentline will be ignored or misread. - Use search-engine testers for Google and Bing rules. Their tools show whether a given URL is blocked for their crawlers. For AI crawlers, there is rarely an official tester, so rely on careful reading and on log checks.
- Watch your logs for a few weeks. Confirm that blocked bots disappear and allowed bots keep arriving. An unexpected drop in Googlebot activity means a rule has gone wrong.
Subdomains need their own robots.txt files, and each protocol and host combination is treated separately. A file at www.example.com does not apply to blog.example.com.
Common mistakes
- Blocking the wildcard by accident. Adding
Disallow: /underUser-agent: *blocks every well-behaved crawler, including Google. Always scope AI blocks to named user agents. - Confusing training with search. Blocking GPTBot does not block ChatGPT search, which uses OAI-SearchBot. Blocking only the training bot and allowing the search bot is a valid and common choice, but make sure it is the one you intended.
- Expecting Google-Extended to hide you from Google Search. It does not control Search or its ranking. It is an AI-use control, and Google documents it separately.
- Looking for control tokens in your logs. Google-Extended and Applebot-Extended never appear as visitors, so you cannot confirm them by watching traffic. Their effect is on how already-crawled content is used.
- Assuming a block is a guarantee. Some crawlers and user-triggered fetchers do not honor robots.txt. If you must prevent access, use firewall or CDN rules.
- Replacing instead of merging. Pasting the generated output over an existing file can delete rules you rely on, such as paths hidden from search engines or your sitemap line. Merge carefully.
- Wrong location or filename. The file must be named
robots.txt, in lowercase, at the root of the host. A copy in a subfolder is ignored. - Blocking pages you want cited. If you disallow AI search crawlers from your product, pricing or help pages, you remove the pages most likely to appear in useful answers.
- Never revisiting the file. Bot names and behaviors change. A file you wrote two years ago may miss the crawlers that matter now.
- Mixing up robots.txt and noindex. Blocking a URL in robots.txt does not remove it from an index, and it prevents crawlers from seeing a noindex tag. They solve different problems.
Blocking, allowing and your WordPress site
On WordPress, robots.txt is often generated dynamically by core or an SEO plugin, which means there may not be a file to edit by hand. The right approach is to use the plugin’s settings or a filter, so your changes are not overwritten on update.
The free aeoora WordPress plugin includes per-bot Allow, Disallow and Block controls with presets and can sync your choices into robots.txt. It also logs which AI crawlers actually visit, so you can see the result of a policy change on your own site. If you use another approach, such as a hand-edited file or a CDN rule, the output from this generator works the same way.
Frequently asked questions
Does robots.txt actually stop AI crawlers?
It stops the ones that choose to obey it. OpenAI, Anthropic, Google, Apple, Perplexity and most major operators state that their main crawlers respect robots.txt. Some user-triggered fetchers and a number of lesser-known crawlers have been reported to ignore it. For a hard block, add server or CDN rules in addition to robots.txt.
If I block GPTBot, will ChatGPT stop mentioning my business?
Not necessarily. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and user requests use ChatGPT-User. Blocking GPTBot only opts out of future training use. The assistant can still talk about you using what it already learned and what it finds on other sites.
Will blocking AI crawlers hurt my Google rankings?
Blocking AI-specific tokens and crawlers such as GPTBot, ClaudeBot or Google-Extended does not change how Googlebot ranks your pages. The risk comes from mistakes, such as a bad wildcard rule. Test carefully and watch Googlebot activity after any change.
What is the difference between Google-Extended and Googlebot?
Googlebot is the crawler that fetches your pages for Google Search. Google-Extended is a robots.txt token that controls whether content Google has crawled may be used for Gemini model training and grounding. It does not crawl on its own and does not affect Search inclusion.
How quickly do changes take effect?
Crawlers cache robots.txt, often for up to a day, and some refresh less often. Expect most compliant bots to pick up changes within a few days. Content that was already collected is not recalled.
Should I block Bytespider?
Many site owners do, because it is widely reported to ignore robots.txt anyway and to collect data for LLM training. If you want to be sure, pair the robots.txt rule with a firewall or CDN rule that blocks the user-agent or its network ranges.
Can I allow AI crawlers on some pages only?
Yes. Use a named group with Disallow lines for the folders you want to protect and an Allow line for the rest. The most specific matching rule wins, so you can allow a subfolder inside a blocked path.
Does this generator store my settings?
No. It runs in your browser and does not upload your choices. If you want to keep a configuration, copy or download the output.
Go further with aeoora
Setting rules is the easy half. Knowing which AI crawlers really visit, how often and which pages they read is the other half. The free aeoora WordPress plugin recognizes more than 300 AI crawlers, lets you set Allow, Disallow or Block per bot and shows a live crawl log you can export. The plugin docs explain how to set it up.
For sites beyond WordPress, aeoora Cloud analyzes your server logs to show verified AI bot traffic, and it tracks how your brand appears in AI answers once crawlers can read your pages. Read about AI crawler analytics, compare plans on the pricing page, or explore more free tools, including the llms.txt Generator.