How to use the AI Bot User-Agent Checker

The AI user agent checker runs in your browser and matches the text you paste against a list of known AI crawlers. Nothing you enter is uploaded or stored.

  1. Open your server logs, CDN dashboard or analytics tool and find a request you want to identify. Copy the full user-agent string, not just part of it.
  2. Paste the string into the input box. You can paste a single line or, if the tool supports it, several lines at once.
  3. Click Check. The tool compares the string with its list of AI crawler signatures.
  4. Read the result. It shows the bot name, the operator (the company behind it), the purpose (training, search indexing or live fetching for a user) and whether the bot is known to respect robots.txt.
  5. If the string is not recognized, the tool tells you so. That can mean it is a normal browser, a search engine, a monitoring script or an AI crawler that is not on the list yet.
  6. Use the result to decide what to do next: allow the bot, block it in robots.txt with the AI Crawler robots.txt Generator, or verify it against the operator’s published IP ranges if the traffic looks suspicious.

Tip: A match tells you what the string claims to be, not who actually sent the request. Treat it as a first identification step and verify important cases using the methods described below.

What a user-agent string is

Every time a browser, crawler or script requests a page, it sends a header named User-Agent that describes the software making the request. Web servers record this header in their access logs, which is why it is the first place to look when you want to know who is visiting.

For a human visitor, the header describes a browser and an operating system. For a crawler, it names the crawler and usually includes a link or email address where a site owner can learn more. The format is not strictly enforced. Historical quirks mean that nearly every string starts with Mozilla/5.0, even when the client is not Mozilla or a browser, because early websites served different content depending on that token.

A crawler’s user-agent often follows the pattern of a product token with a version number, such as GPTBot/1.1, along with a compatibility note and a documentation URL. The product token is what the checker looks for, and it is also the token you use in robots.txt in most cases.

Why identifying AI bots matters

For a long time, bot traffic meant search engines, a handful of monitoring services and spammers. Today a large and growing share of automated requests comes from AI companies, and they serve very different purposes. Knowing who is visiting helps with several decisions.

  • Visibility. If you want AI assistants to describe your business accurately, their crawlers need to reach your pages. Seeing which bots have visited tells you whether you are on their radar.
  • Content control. If you do not want your content used for model training, you need to know which bots collect training data, and you need to recognize them in your logs.
  • Capacity and cost. Aggressive crawlers can load your servers, inflate bandwidth bills and distort analytics. Identifying the source is the first step to managing it.
  • Analytics accuracy. Automated traffic can skew page-view counts, conversion rates and engagement metrics if it is not filtered out.
  • Security. Fake bots that impersonate legitimate crawlers are a common way to scrape content or probe a site. Identification helps you spot the impostors.

Three kinds of AI visitor

The most useful thing the checker tells you is not only the name of a bot, but its purpose. AI visitors fall into three broad groups, and each one calls for a different response.

Training crawlers

Training crawlers collect pages in bulk so the content can be used to build or improve language models. They behave like traditional crawlers: they follow links, request many pages and come back periodically. Examples include GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider and Amazonbot. If you object to your content being used in training, these are the bots to block. Blocking them does not make your pages vanish from AI search.

Search and index crawlers

These crawlers build the index that powers an AI search product. They fetch pages so that an assistant can later find them, link to them and cite them. Examples include OAI-SearchBot, Claude-SearchBot and PerplexityBot. If you want to appear in AI answers, you generally want these bots to be able to read your site. Operators describe them as separate from their training crawlers.

User-triggered fetchers

These visit a page at the moment a person asks an assistant about it. Examples include ChatGPT-User, Claude-User and Perplexity-User. They produce small bursts of traffic that reflect real questions. A visit from one of these bots means a real person asked an AI about your page or topic, which is valuable information. They often behave more like a browser than a crawler, and some operators indicate their robots.txt handling differs from that of their main crawlers.

There is also a fourth group worth knowing: AI agents that browse the web to complete tasks for a user, such as ChatGPT Agent or Claude Code’s web fetch. They may identify themselves differently, sometimes through a signed header rather than the user-agent string, which makes them harder to spot by name alone.

What the checker can tell you

When a string matches a known signature, the result includes several fields. Here is how to read each one.

  • Bot name. The name the operator uses, such as GPTBot or PerplexityBot. This is also the token for robots.txt rules in most cases.
  • Operator. The company that runs the bot, for example OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance or Common Crawl.
  • Purpose. Whether the bot is for training, search indexing, live fetching for users or agent browsing.
  • Robots.txt behavior. Whether the operator says the bot respects robots.txt. Some entries are marked as unknown or as bots that may not follow the file.
  • Documentation. A link to the operator’s own page about the bot, which is the authoritative source for IP ranges and opt-out instructions.

The list behind the checker follows the same approach as the free aeoora WordPress plugin, which recognizes more than 300 AI crawlers and groups them by operator and purpose. New bots appear often, so an unrecognized string is not proof that something is human.

How user-agent matching works

The checker looks for known substrings in the string you paste. For example, if it contains GPTBot, it is classified as OpenAI’s training crawler. The match is not case-sensitive, so claudebot and ClaudeBot are treated the same.

Order matters. Some operators run several bots whose names overlap. Anthropic’s ClaudeBot, Claude-SearchBot and Claude-User all start with the same letters, and OpenAI’s ChatGPT-User differs from its ChatGPT Agent. A good matcher checks the more specific names before the general ones, so a Claude-SearchBot request is not mislabeled as ClaudeBot.

Some bots do not have a user-agent string you can match at all. Google-Extended and Applebot-Extended are robots.txt control tokens and never appear in your logs. Their parent crawlers, Googlebot and Applebot, do appear. If you see Googlebot in your logs, you cannot tell from the string alone whether that fetch will be used for traditional search, for AI features or both.

The limits of user-agent identification

User-agent strings are easy to read and easy to fake. Before you act on a result, keep these limits in mind.

First, anyone can send any user-agent. A scraper can claim to be GPTBot to get past a rule that allows it, or claim to be Googlebot to avoid being blocked. The string is a label, not proof.

Second, some AI traffic does not identify itself clearly. Agents that drive a real browser may look like ordinary Chrome sessions. Some third-party tools that fetch pages for AI products use generic HTTP library strings.

Third, a visit from an AI bot does not tell you that your content was used in an answer. Crawling and citing are separate steps, and a crawler can visit pages that never appear in a response.

Fourth, bots change. Operators rename crawlers, retire old ones and split one bot into several. Treat any list, including this one, as a snapshot that needs periodic updating.

How to verify a bot is genuine

When it matters, such as before allowing a bot through a firewall or blaming a crawler for a traffic spike, verify the visitor beyond its user-agent. There are three common approaches.

Check published IP ranges

Several operators publish lists of the IP addresses their crawlers use. OpenAI, Perplexity and others provide machine-readable lists, and Google and Bing document how to verify their crawlers. Compare the source IP from your log with the operator’s list. A request that claims to be GPTBot but comes from an address outside OpenAI’s ranges is not GPTBot.

Use reverse and forward DNS

For crawlers whose operators support it, perform a reverse DNS lookup on the IP to get a hostname, check that the hostname belongs to the operator’s domain, then do a forward lookup on that hostname to confirm it resolves back to the same IP. This method is documented by Google for Googlebot and by Bing for Bingbot.

Use signed requests where available

A newer approach is for agents to sign their requests cryptographically, so that a site can verify the sender without maintaining IP lists. Some AI agents now send signature headers for this purpose. If your CDN supports verified bots, it may check these for you.

For large volumes of logs, doing this by hand is impractical. A log-analysis tool that checks both the string and the source address will separate real crawlers from impostors far more reliably.

Best practices

Capture the full string

Partial strings can lead to wrong matches. Copy the entire header from the log line, including version numbers and URLs. The documentation link in the string is often the clearest sign of who is behind the request.

Identify before you block

Look up what a bot does before you add a rule. Blocking a search crawler by mistake removes your pages from that search product, and blocking a user-triggered fetcher makes the assistant unable to read a page a person asked about.

Combine the string with IP verification

Use the checker to name the bot, then verify the IP for any bot you plan to treat specially. This protects you from both false trust and false accusations.

Look at patterns, not single requests

One request tells you little. Look at which paths a bot requests, how often and over what period. Training crawlers tend to request many pages steadily. User-triggered fetchers produce isolated requests for specific pages. A sudden flood from a single claimed bot may signal an impostor.

Keep a record of decisions

Write down which bots you allow, which you block and why. When a vendor changes its crawlers, you will know which rules to revisit.

Feed what you learn back into robots.txt

Identification is most useful when it leads to a policy. Once you know which bots are visiting, set explicit rules with the AI Crawler robots.txt Generator and confirm the effect in your logs.

Watch for referrals as well as crawls

A crawl and a visit from a person following an AI citation are different events. Crawlers show up in server logs with a bot user-agent. People arriving from an assistant show up in your analytics as referral traffic from sites such as chatgpt.com or perplexity.ai. Tracking both gives you the full picture.

Examples

These are illustrative user-agent strings for common AI bots. Real strings vary by version and change over time, so use the operator’s documentation to confirm the current format. Paste any of them into the checker to see how results look.

Training crawler example

This string is typical of OpenAI’s training crawler. The checker would report GPTBot, operated by OpenAI, purpose training, and note that it respects robots.txt.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

Search crawler example

This is the kind of string used by Perplexity’s index crawler. The checker would report PerplexityBot, operated by Perplexity, purpose search indexing.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

User-triggered fetch example

This resembles the string sent when a ChatGPT user asks about a page. The checker would report ChatGPT-User, operated by OpenAI, purpose live fetching for a user, and flag that robots.txt may not apply to it.

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Spotting AI bots in a log

In a standard access log, the user-agent is the last quoted field. You can filter for common AI crawlers with a short command on a Linux server.

grep -Ei "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-externalagent|Amazonbot" access.log | awk '{print $1, $7}' | head

This prints the source IP and the requested path for the first matching lines. From there you can count requests per bot, or compare the IPs against the operator’s published ranges.

A miss that is still informative

If you paste a string like the one below, the checker will say it is not a known AI crawler. It is a regular desktop browser.

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36

A plain browser string does not prove a human sent it, only that the client chose to look like a browser.

Common mistakes

  • Trusting the label. A user-agent can be copied by anyone. Do not whitelist or blame a bot on the string alone when the stakes are meaningful.
  • Assuming every AI bot is a trainer. Many visits come from search crawlers and user-triggered fetchers. Blocking them all removes you from AI answers.
  • Searching logs for control tokens. Google-Extended and Applebot-Extended never appear as visitors, so an empty search does not mean they are being ignored.
  • Matching on partial words. Searching for “Claude” will mix up ClaudeBot, Claude-SearchBot, Claude-User and unrelated strings. Match the full product token.
  • Ignoring unrecognized strings. A new AI crawler will not be recognized until lists catch up. If a string includes the words bot, crawler or spider and a link to a documentation page, read the page before dismissing it.
  • Blocking by user-agent alone. Determined scrapers rotate strings. Rate limits, IP checks and bot-management tools are more durable.
  • Treating crawls as citations. A bot visiting a page does not mean it appeared in an answer. Measure visibility separately.
  • Forgetting user-triggered visits are meaningful. A ChatGPT-User or Perplexity-User request often means a real person asked about that page. Check which pages draw these requests.
  • Not counting AI traffic separately. If you lump AI crawlers with all bots, you cannot see trends in how often AI systems read your site.
  • Setting rules and never checking them. After changing robots.txt, confirm in the logs that the behavior changed.

Reading AI bot activity as a signal

Beyond defense, AI crawler logs are a source of insight. Which pages do the search crawlers request most often? Those are the pages AI systems consider important enough to index. Which pages do user-triggered fetchers request? Those are the pages real people are asking assistants about. A page that gets visits from fetchers but little traffic from search may be answering a question that AI users care about.

Compare the data against your goals. If your most commercial pages are never fetched by any AI bot, check whether robots.txt, a login wall, a heavy JavaScript layout or a firewall rule is stopping them. If a crawler hits thousands of low-value URLs, such as filtered product listings or calendar pages, consider disallowing those paths so crawl effort goes to the content that matters.

Over time, a record of AI bot activity also shows how quickly new content gets discovered. If you publish an article and a search crawler requests it within hours, your site is well connected. If it takes weeks, improve internal linking, update your sitemap and consider notifying search engines through protocols such as IndexNow, which some engines use.

Frequently asked questions

Which AI bots does the checker recognize?

It recognizes the main AI crawlers and assistants, including GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User from Perplexity, plus bots from Meta, Amazon, ByteDance, Common Crawl, Mistral, DuckDuckGo, You.com and Cohere. The list grows as new crawlers appear.

Can the tool prove a request really came from OpenAI or Anthropic?

No. It identifies what the string claims. To prove the source, compare the request’s IP address with the operator’s published ranges or use DNS verification where the operator supports it.

What is the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI’s crawler for collecting content that may be used to train models. ChatGPT-User fetches a specific page when a user of ChatGPT asks about it. The first is bulk collection, and the second is on-demand reading for a person.

Why does my log show Googlebot but never Google-Extended?

Google-Extended is a robots.txt control token, not a crawler. It tells Google how it may use content that Googlebot already crawled. It has no user-agent string and does not appear in logs.

Should I block every AI bot I find?

Not by default. Decide by purpose. Many site owners block training crawlers and allow search and user-triggered fetchers so they remain eligible to be cited. Your own goals decide the right mix.

What if the checker says my string is unknown?

It might be a browser, a search engine, a monitoring tool or a new AI crawler. Look for a URL or email in the string and read the documentation it points to. If you cannot find any, treat it with caution and check the IP address.

Does the tool store the strings I paste?

No. The matching runs in your browser, and your input is not sent anywhere or saved.

How often does the list of bots change?

Fairly often. AI companies launch new crawlers, rename existing ones and separate training from search. Revisit your findings every few months and update your rules when vendors announce changes.

Go further with aeoora

Checking one string at a time is useful for investigation. If you want ongoing visibility into AI bot traffic, the free aeoora WordPress plugin recognizes more than 300 AI crawlers, logs their visits, lets you export a crawl log and sets per-bot rules. It also tracks visits that arrive from AI assistants without cookies. See the plugin documentation to get started.

If your site is not on WordPress, or you want verified bot data across your whole stack, aeoora Cloud analyzes server logs with user-agent and IP range verification, and connects crawler activity to how your brand appears in AI answers. Learn how it works in the AI crawler analytics docs, see the pricing, or return to the other free tools.