Overview
Your site is visited by a growing list of AI crawlers. Some collect content to train models. Some fetch pages so an assistant can answer a question right now. Some act on behalf of a person who asked an assistant to look something up. Treating them all the same, by blocking everything or allowing everything, usually gets the trade-off wrong.
This guide explains the main categories, how to decide what to allow, how to implement the policy safely, and how to confirm the bots you see are the bots they claim to be.
The main kinds of AI bots
Operators describe their crawlers differently, but most fall into a few groups.
- Training crawlers: collect public content to build or improve models. Allowing them contributes to model knowledge, but it does not by itself earn you a citation in a live answer.
- AI search crawlers: build an index that an AI search product uses to find and cite pages. Blocking these can remove you from that product’s answers.
- User-triggered fetchers: retrieve a page when a person asks an assistant a question that needs it. These are the closest thing to a live visit from an AI user.
- AI agents: browse and act for a user, for example to compare products or fill a form.
- Everything else: search engine bots, SEO tools, social previewers and scrapers also appear in your logs. They are not AI crawlers in the strict sense, but you will want to tell them apart.
The names and behaviors of individual bots change as operators release new ones, so a fixed list goes stale. What stays useful is the category.
Decide what you want
The right policy depends on your business. Ask what you want from AI systems.
If you want to be recommended and cited
Allow AI search crawlers and user-triggered fetchers. These are the bots that decide whether your pages can appear in an answer. Most brands, publishers with an audience goal, SaaS companies and local businesses fall here.
If you worry about training use
You can block training crawlers while still allowing search and fetch bots. Be aware that policies differ by operator, and blocking a training crawler is a statement of preference, not a technical guarantee.
If your content is paid or sensitive
Gate it properly with authentication rather than relying on robots.txt. Robots.txt is a request that well-behaved bots honor, not a security control.
If your servers are under strain
Check how much traffic AI bots really send before deciding. If a particular bot is aggressive, you can slow it with rate limits instead of blocking it outright.
Tip: Review your policy per bot, not per company. One operator may run a training crawler, a search crawler and a user fetcher with separate names, and you can treat each differently.
How to implement your policy
There are three layers, and they work together.
- robots.txt: the standard place to allow or disallow named bots. It is public, easy to audit, and respected by the major operators. Use specific user-agent entries for the bots you care about.
- Server or CDN rules: a firewall or edge rule can return an error to a bot that ignores robots.txt. Use this sparingly and never on generic clients, login pages, REST endpoints or checkout flows.
- Content signals and llms.txt: emerging conventions let you express how content may be used, and a short llms.txt file points assistants to the pages you most want read. They are helpful hints rather than guarantees.
Keep the file simple. A long list of rules is hard to maintain, and a typo can accidentally block everything.
Check that your rules really work
Policies fail quietly. Common problems include:
- A CDN or security plugin that blocks AI bots even though robots.txt allows them.
- A page cache that serves a stale robots.txt or ignores your block rules.
- A staging robots.txt that went live with a blanket disallow.
- Another plugin or app that also serves llms.txt or robots.txt and overrides yours.
- Rules written for one name when the operator publishes several.
Fetch your live robots.txt from outside your network and compare it with what you expect. Then look at your logs to see what the bots do.
Verify who is really visiting
Anyone can put an AI company’s name in a user-agent string. If you act only on the user-agent, you may allow impostors or block the real thing. Better practice is to verify bots by combining the user-agent with the operator’s published IP ranges, or with signatures that the operator documents. Real traffic from verified bots tells you which assistants are reading you, how often, and which pages they fetch.
Server logs are the source of truth here. Browser analytics tools do not see crawlers, because bots do not run your analytics script.
Read the results
Once you have a few weeks of data, look for patterns.
- Which pages do AI bots fetch most? These are your de facto AI landing pages. Keep them accurate and fast.
- Which important pages do they skip? Check for blocks, weak internal links, missing sitemap entries or slow responses.
- Are there errors? A steady stream of 403 or 5xx responses to a bot you want means a configuration problem.
- Are impostors present? Unverified traffic using a famous name deserves a closer look.
- Does crawl activity line up with citations? Pages that bots fetch and that answer questions well are the ones most likely to be cited.
A sensible default
If you have no strong reason to restrict, a good starting point is to allow AI search crawlers and user-triggered fetchers, make an explicit decision about training crawlers, leave generic clients and your application endpoints alone, and review the list quarterly. Document why each rule exists so the next person does not undo it by accident.
How aeoora helps
The free aeoora WordPress plugin recognizes 325 AI crawlers, lets you set Allow, Disallow or Block per bot with presets, syncs robots.txt, and warns you when a cache or another plugin overrides your rules. It also generates llms.txt and Markdown versions of your pages. For sites on any platform, aeoora Cloud analyzes verified AI bot traffic from your logs through Cloudflare, Vercel, a webhook or log upload, on the Signal and Scale plans.
Learn more in the docs, compare plans on the pricing page, or start a free trial.


