How to use the Robots.txt Tester

This free robots.txt tester tells you whether a specific crawler may fetch a specific URL on your site, and it shows the exact rule that made the decision. It runs entirely in your browser. You paste the file yourself, nothing is fetched from your server, and nothing you paste is sent anywhere.

  1. Copy your robots.txt. Open https://yoursite.com/robots.txt in a browser and copy all of the text. Because the tester does not fetch the file, you can just as easily paste a draft that is not live yet.
  2. Paste it into the tester. Paste the whole file, comments included, so the line numbers in the results match your file.
  3. Enter a URL or path. A full address such as https://yoursite.com/blog/my-post/ or a path such as /blog/my-post/ both work. Robots.txt rules only look at the part after the domain.
  4. Choose a user agent. Pick a crawler from the list: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent or Bytespider. If yours is not listed, type it as a custom user agent.
  5. Read the verdict. You get Allowed or Blocked, plus the rule that decided it, its line number and the user-agent group that applied.
  6. Check the All AI bots matrix to see the result for every listed bot on the same path at once.
  7. Review the warnings and the Sitemap lines the tester found, and fix anything that looks wrong.

Tip: Start with the pages that matter most: your home page, a key product or service page, your pricing page and a blog post. If those come back as expected for Googlebot and the AI crawlers you care about, you have covered most of the risk.

What the tester shows you

A robots.txt tester is only useful if it explains its answer. Each part of the result answers a different question about your file.

The verdict and the rule behind it

Every test ends in Allowed or Blocked. Next to the verdict, the tester shows the single rule that decided it, such as Disallow: /checkout/, the line where that rule sits and the user-agent group it belongs to. The group matters more than it sounds. A rule can look as if it applies to every bot when it sits in a group the bot you picked never reads.

When no rule in the applicable group matches the path, the URL is allowed. Robots.txt only blocks what it names, so silence means permission.

The All AI bots matrix

The matrix runs the same path against every bot in the tester’s list and shows Allowed or Blocked for each. It is the quickest way to check a policy such as “search crawlers in, training crawlers out”, and to catch the bot that slipped through because its group was misspelled or never added.

Warnings and sitemap lines

The tester warns about rules placed before any User-agent line, unknown directives, Crawl-delay lines (which Google ignores), relative sitemap URLs, a blanket Disallow: / and an empty file. Each is explained under common mistakes below. It also lists the Sitemap lines it finds, so you can confirm they point at the right files. Sitemap lines belong to no group and apply wherever they appear.

How robots.txt matching works

Every crawler that honors robots.txt makes two decisions: which group of rules applies to it, then which rule in that group matches the URL. The tester follows Google’s documented rules, which were also published as an internet standard, RFC 9309, in 2022.

Step one: pick the user-agent group

A robots.txt file is made of groups. Each group starts with one or more User-agent lines and continues with the Allow and Disallow rules beneath them. A crawler looks for the group whose user agent matches its own name most specifically. If it finds one, it uses that group and ignores every other group. If it finds none, it falls back to the User-agent: * group. If there is no * group either, nothing is blocked.

The consequence surprises many site owners. The * group is a fallback, not a base layer. Once a bot has a group of its own, the rules under * no longer apply to it at all.

Groups for the same user agent are merged

If the same user agent appears in two separate groups, for example a GPTBot section at the top of the file and another at the bottom, the crawler combines them and applies all of their rules together. Several User-agent lines in a row also share the rules that follow them, which is a tidy way to give several bots the same policy.

Step two: the longest matching rule wins

Within the chosen group, the crawler compares the URL path with every Allow and Disallow rule. Rules match from the start of the path, so Disallow: /private matches any path that begins with /private. When several rules match, the one with the longest path wins, because a longer path is a more specific instruction. The order of the rules in the file does not matter.

If an Allow rule and a Disallow rule match with paths of exactly the same length, Allow wins. When your file is ambiguous, crawlers lean toward access. An exact tie is usually an editing mistake, so treat it as a prompt to delete one of the lines.

Wildcards: the asterisk and the dollar sign

An asterisk (*) matches any sequence of characters, including none. A dollar sign ($) at the end of a rule anchors it to the end of the URL. Together they let you write rules for patterns rather than folders, such as every URL that ends in .pdf. A trailing asterisk adds nothing, because rules already match by prefix: /private* behaves exactly like /private.

What is case-sensitive and what is not

Paths are case-sensitive, so Disallow: /Admin/ does not block /admin/. User-agent names work the other way: gptbot and GPTBot are the same bot, and two groups that differ only in capitalization are merged. Directive names such as Disallow are not case-sensitive either.

Worked examples

Each example below describes a short robots.txt file in words, then the decision a crawler makes under these rules, which is the decision the tester shows you.

A bot with its own group ignores the wildcard group

Picture a file with two groups. The first is User-agent: * with Disallow: /private/. The second is User-agent: GPTBot with Disallow: /drafts/. Test /private/report for ClaudeBot and the answer is Blocked: ClaudeBot has no group of its own, so it falls back to the * group, where the /private/ rule matches.

Test the same path for GPTBot and the answer is Allowed. GPTBot has its own group, so it ignores the * group entirely, and its group only mentions /drafts/. To keep GPTBot out of /private/ too, repeat that rule inside the GPTBot group. The group shown next to the verdict makes this common surprise obvious.

An open subfolder inside a blocked folder

Suppose the * group contains Disallow: /shop/ and Allow: /shop/sale/. For /shop/sale/winter-boots, both rules match. The Allow path is eleven characters long and the Disallow path is six, so Allow wins and the URL is allowed. For /shop/cart, only the Disallow rule matches, so it is blocked. Swap the order of the two lines and you get the same answers.

Blocking file types with the dollar sign

The rule Disallow: /*.pdf$ blocks any URL that ends in .pdf, in any folder, so /files/price-list.pdf is blocked. A URL such as /files/price-list.pdf?download=1 is not, because the dollar sign requires the URL to end right after .pdf. Drop the dollar sign and the rule matches both, along with any other path that contains .pdf after a slash.

A missing slash that blocks too much

Disallow: /blog blocks the blog folder, but it also blocks /blog-archive, /blogging-tips and /blog.html, because rules match by prefix. If you only want the folder, write Disallow: /blog/. Testing a couple of neighboring paths, not just the one you had in mind, catches this kind of overreach.

AI crawlers: training, search and user-triggered fetchers

The AI crawlers in the tester’s list do very different jobs, and blocking one is not the same as blocking another. Sort them by purpose before you decide what your file should say.

Training crawlers

GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), meta-externalagent (Meta) and Bytespider (ByteDance) collect content that may be used to train or improve AI models. CCBot builds the open Common Crawl archive, a dataset many organizations use for training. Blocking these crawlers is how site owners opt out of future training collection.

AI search crawlers

OAI-SearchBot (ChatGPT search), Claude-SearchBot (Claude) and PerplexityBot (Perplexity) build the indexes that AI search features use to find, link to and cite pages. OpenAI, Anthropic and Perplexity describe these bots as separate from their training crawlers, so you can allow one and block the other. Googlebot and Bingbot belong in this family too, and Google’s AI features in Search, such as AI Overviews, draw on what Googlebot crawls.

User-triggered fetchers

ChatGPT-User, Claude-User and Perplexity-User visit a page live when a person asks an assistant about it, for example by pasting a link. They behave more like a browser working for one person than a crawler. Vendors differ on whether robots.txt applies here. Anthropic says Claude-User respects robots.txt, while OpenAI and Perplexity have indicated that their user-initiated fetchers may not follow it in the same way.

Control tokens

Google-Extended and Applebot-Extended are not crawlers. They are names you use in robots.txt to tell Google and Apple how they may use content their regular crawlers already collect. Neither token appears in your server logs, so a robots.txt tester is the simplest way to confirm what your file tells them.

What blocking each type does to AI search visibility

  • Blocking training crawlers keeps your pages out of future training collection by those companies. It does not remove what a model already learned, and it does not stop that company’s search product from citing you, as long as its search crawler is allowed.
  • Blocking AI search crawlers costs visibility most directly. If OAI-SearchBot, Claude-SearchBot or PerplexityBot cannot crawl a page, that product has a much harder time surfacing and citing it. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links.
  • Blocking user-triggered fetchers can mean an assistant fails to read your page at the moment someone asks about it, so it answers from other sources. The effect differs by vendor.
  • Blocking Google-Extended opts your content out of Gemini model training and grounding. Google says it does not affect inclusion in Google Search, and AI Overviews draw on Googlebot’s crawl rather than this token. Blocking Googlebot itself removes you from both.
  • Blocking Applebot-Extended opts your content out of Apple’s foundation model training. Applebot keeps crawling for Siri and Spotlight.

A common middle ground is “search yes, training no”: allow the search crawlers and user-triggered fetchers, and block the training crawlers and the two control tokens. The AI Crawler robots.txt Generator builds that kind of policy, and this tester confirms it does what you intended. Assistants can still describe your brand from other sites, so blocking your own site mostly removes your most accurate voice.

Common robots.txt mistakes

Most broken robots.txt files fail in a handful of predictable ways. The tester warns about several of them directly, and the rest show up as soon as you test the right path.

  • Assuming the * group covers everyone. A bot with its own group ignores the * rules. If a rule should apply to every bot, repeat it in each named group.
  • Shipping a staging file. A blanket Disallow: / under User-agent: * is normal on a staging site and a disaster in production. The tester warns about it every time, so you can confirm it is deliberate.
  • Rules with no group. Allow or Disallow lines above the first User-agent line belong to no group, and crawlers skip them.
  • Typos and unsupported directives. A misspelled directive such as Dissallow is unreliable: some crawlers forgive common typos and others skip the line. A Noindex line does nothing for Google, which stopped supporting it in robots.txt in 2019.
  • Relying on Crawl-delay for Google. Google ignores Crawl-delay, although some other crawlers, including Bingbot, honor it. If Googlebot is putting too much load on your server, follow Google’s own guidance on reducing its crawl rate.
  • Relative sitemap URLs. Sitemap: /sitemap.xml should be Sitemap: https://yoursite.com/sitemap.xml. Google expects a fully qualified URL.
  • Blocking CSS and JavaScript. Google renders pages much as a browser does. If robots.txt blocks the stylesheets or scripts a page needs, Google may not see the page the way visitors do. Test a few asset URLs as well as page URLs.
  • Using robots.txt to hide secrets. The file is public. Listing /internal-reports/ tells anyone who reads it exactly where to look. Protect private content with a login.
  • An empty or overwritten file. An empty robots.txt allows everything. If you expected rules and the tester reports an empty file, a deploy or a plugin may have replaced your real one.

Robots.txt vs noindex vs X-Robots-Tag

These three are often treated as interchangeable, and they are not. Robots.txt controls crawling. The noindex meta tag and the X-Robots-Tag header control indexing and how a page is shown. Mixing them up is how pages stay in search results you wanted them out of.

Robots.txt controls crawling

A Disallow rule asks a crawler not to request a URL. It does not ask a search engine to forget it. Google says a page blocked by robots.txt can still be indexed if other pages link to it, in which case it may appear without a description. Robots.txt is the right tool for keeping crawlers out of whole sections, managing crawl load and setting a policy for individual AI crawlers.

The noindex meta tag controls indexing

A tag such as <meta name="robots" content="noindex"> tells search engines not to show a page in results. The crawler has to fetch the page to see the tag, so if you also block the page in robots.txt, the noindex is never read and the page can linger in the index. To remove a page from results, leave crawling allowed and add noindex.

X-Robots-Tag does the same job in an HTTP header

X-Robots-Tag carries the same directives, such as noindex and nosnippet, in the HTTP response instead of the HTML. That makes it the only option for PDFs, images and other files without a head section, and a convenient one for applying a directive to many URLs from your server configuration. It also only works if the crawler may fetch the URL.

Which one to use

  • Keep a crawler out of a section, or set a policy for one AI bot: robots.txt.
  • Keep an HTML page out of search results: a noindex meta tag, with crawling allowed.
  • Keep a PDF or other file out of search results: an X-Robots-Tag header.
  • Limit how much of a page appears in a snippet: nosnippet or max-snippet. Google points to these same controls for limiting what its AI features in Search show from your pages.
  • Keep content genuinely private: a login. None of these mechanisms is access control.

For AI crawlers outside Google, robots.txt is the control vendors document most consistently. Support for page-level directives varies, so check each vendor’s documentation before relying on a meta tag.

Test changes before you deploy with the robots.txt tester

A one-character mistake in robots.txt can block an entire site. Google generally caches robots.txt for up to 24 hours, so a bad file can shape crawling for a day before a fix is seen. A few minutes with a robots.txt tester before each deploy is cheap insurance.

  1. List your test URLs. Include the home page, key product, service and pricing pages, a blog post, a URL with a query string, and one CSS and one JavaScript file. Add every path your change is meant to block.
  2. Test the live file first. Paste your current robots.txt and run each URL for Googlebot, Bingbot and the AI crawlers you care about. Note the results as your baseline.
  3. Paste the draft and test again. Make your change in a text editor, paste the whole draft and rerun every URL. Only the URLs you meant to change should flip between Allowed and Blocked.
  4. Check the matrix and the warnings. The matrix catches a new group that quietly pulled a bot away from the * rules. Clear every warning, and confirm that any blanket Disallow: / is intentional.
  5. Deploy, then check the live file. Open https://yoursite.com/robots.txt and paste what you see back into the tester. Caches, CDNs and plugins sometimes serve something other than what you uploaded. On WordPress, robots.txt is often generated by WordPress or a plugin rather than read from a file.
  6. Watch real crawler activity. Confirm over the next few days that blocked bots stop requesting the paths you closed and allowed bots keep arriving. Google Search Console’s robots.txt report shows the version Google last fetched, and aeoora’s AI crawler analytics shows which verified AI bots read which pages.

Each host needs its own file. A robots.txt at www.example.com does not apply to shop.example.com, so test the file for every host you serve.

Frequently asked questions

Does the robots.txt tester fetch my live file?

No. You paste the contents yourself, and the tester reads them in your browser. Nothing is fetched from your server and nothing you paste is sent anywhere, so you can safely test drafts that mention unpublished paths.

Will Google treat my file exactly as the tester does?

The tester follows Google’s documented rules and RFC 9309, so for a well-formed file you should get the answer Google would give. Real crawling can still differ: Google may be using a cached copy, or your server may serve a different file from the one you pasted. Other crawlers may handle wildcards or conflicting rules differently.

Why is a URL allowed when I have a Disallow rule for it?

Usually one of four reasons. The bot has its own group, so your rule under * does not apply to it. A longer Allow rule matches the same path. The case of the path differs from the rule. Or the rule does not match the start of the path. The deciding rule and group shown in the result point straight at the cause.

Does the order of rules matter?

Not within a group. The longest matching path wins wherever it sits. Order matters only because rules belong to the User-agent lines above them. Some older parsers used the first matching rule instead, so an unambiguous file protects you against them as well.

If I block GPTBot, will ChatGPT stop mentioning my site?

Not necessarily. GPTBot is OpenAI’s training crawler. ChatGPT search relies on OAI-SearchBot, and live fetches on a user’s request come from ChatGPT-User. Blocking GPTBot alone keeps your pages out of future training, while an allowed OAI-SearchBot keeps them eligible to be cited in ChatGPT search. The matrix shows all three at once.

Will blocking AI crawlers affect my Google rankings?

Blocking crawlers such as GPTBot, ClaudeBot or PerplexityBot has no effect on Google Search, and Google says blocking Google-Extended does not affect inclusion in Search either. Blocking Googlebot is different: it stops Google from crawling your pages, which affects both regular results and AI Overviews.

Do all crawlers obey robots.txt?

No. Robots.txt is voluntary. Major search engines and the main AI companies say their crawlers follow it, with the exceptions noted above for some user-triggered fetchers. Some crawlers, Bytespider among them, are widely reported to ignore it, and anyone can fake a user-agent string. Check your server logs, and use the AI Bot User-Agent Checker to identify the bots you find there.

What should I type as a custom user agent?

Type the short name the crawler looks for in robots.txt, such as Amazonbot or Applebot, rather than the full user-agent string from your logs. Robots.txt groups are written against those short names, and each crawler’s documentation lists the one it uses.

What happens if my robots.txt is empty or missing?

Both mean crawlers may fetch everything. An empty file has no rules, and a robots.txt that returns a 404 is treated as having none. The tester warns about an empty file because it often means something overwrote your real rules. A robots.txt that returns a server error is different: Google may pause crawling the site until the file can be fetched again.

Robots.txt decides whether crawlers can reach your pages. These free tools help with what comes before and after.

Browse the full set, including schema generators, on the free tools hub.

Go further with aeoora

A correct robots.txt makes sure the right crawlers can reach your pages. It cannot tell you whether AI assistants actually mention or cite you once they do. The free AI visibility report checks that for your brand across 25 buyer questions and emails you a PDF.

To keep watching after that, start a 2-day trial of aeoora Cloud. It tracks how your brand appears in answers from ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews, and its AI crawler analytics shows which verified bots from OpenAI, Anthropic and Perplexity read your pages, so you can confirm your robots.txt changes work in practice. You can compare plans on the pricing page.