Ai Search
AI Crawler User Agents: The Reference Table
An AI crawler user agent is the identifying string a bot sends in its HTTP request header when it fetches your pages for an AI system. This list covers the ones that matter now, what each one actually does, and which you want to allow so an assistant can read and cite you.
There is no single official registry. Operators publish their own agents on their own documentation, run more than one, and rename them without notice. The list below is grouped by the company behind the bot and by the job the bot does, because that job decides whether blocking it protects you or quietly erases you from AI answers.
The three jobs an AI crawler does
Before the names, the categories. Almost every mistake businesses make with AI crawlers comes from treating many different agents as one thing to allow or block. They do three distinct jobs, and the job is what matters.
Most owners block the first group to protect content and accidentally block the second and third with the same rule. That is how a site vanishes from ChatGPT and Perplexity while its owner believes they only opted out of training.
The AI crawler user agents list
OpenAI
Anthropic (Claude)
Perplexity
Others worth knowing
User agent versus robots.txt token
Two different things get called AI crawlers, and mixing them up produces rules that do nothing. A user agent is a string sent in a real request, and it shows up in your server logs when the bot visits. A product token like Google-Extended or Applebot-Extended is a name you can address in robots.txt that no bot ever sends as a visitor. It exists only so you can set a policy.
Blocking by token and blocking by user agent are both valid, but they are not interchangeable. A rule aimed at a token that expects a visiting agent, or the reverse, silently does nothing. And robots.txt itself is a request. Major operators honor it voluntarily, it is not enforced, and it governs crawling going forward rather than data a model already trained on.
Which agents to allow, and why blocking is a trade
The decision that affects whether you get cited lives in the retrieval and user-fetch group. Allow those and an assistant can read your live pages, verify a claim, and attribute the answer to you. Block them and you opt out of the citation entirely, no matter how strong your content is.
Training crawlers are a real judgment call rather than an obvious block. Keeping GPTBot or CCBot out holds your words back from future training sets. It does not remove you from a model that already learned from you, and on its own it does not touch live retrieval.
We set this up while building the site, because crawler access is wired into robots.txt, the server, and any CDN rules sitting in front of it. A clean answer and a good link do nothing if an aggressive bot rule two layers up returns a 403 to the one agent that would have cited you. That coordination across the stack is the part a content checklist cannot reach.
The honest way to know what is actually reaching your site is your own server logs, not a vendor dashboard. Every agent in this list shows up there by name when it visits. A week of logs tells you which assistants are fetching you, which are being turned away at the door, and whether a rule you thought you set is doing what you intended.
Why this list is never final
Any AI crawler reference is a snapshot. OpenAI shipped a distinct search bot after GPTBot was already in the wild. Anthropic split its single crawler into separate training, search, and user agents. Perplexity added a user-triggered fetcher once live browsing became a feature. The pattern holds: as each assistant separates training from retrieval, it adds agents, and a robots.txt written against last year's names quietly leaves gaps.
Read the list by that shape rather than by memorizing strings. Nearly every serious operator now runs something like a training crawler, a search crawler, and a user fetcher. Once you see that structure, a new agent name tells you its job, and you know whether allowing it helps your visibility or only feeds a future model. Treat any list, including this one, as a starting point you verify against each operator's current documentation, and recheck it on a schedule rather than once.
Questions
Where is the official list of AI crawler user agents?
There is no central registry. Each operator publishes its own agents on its own documentation, and adds or renames them without notice. The practical move is to track the handful of major operators and recheck their robots pages on a schedule rather than trusting a static list.
Is GPTBot the same as the crawler that lets ChatGPT cite me?
No. GPTBot gathers training data. The agents that decide whether ChatGPT can surface and link you are OAI-SearchBot and ChatGPT-User. Blocking GPTBot while allowing those keeps you citable but out of future training.
Does Google use a separate crawler for AI Overviews?
No. Googlebot's normal index feeds AI Overviews. Google-Extended is a robots.txt control token for Gemini training and grounding, not a bot that visits your site, so there is no AI Overviews crawler to allow separately.
If I block AI crawlers in robots.txt, is my content safe?
Only partly. robots.txt is honored voluntarily by the major operators and governs crawling going forward. It does not remove content a model already trained on, and smaller or non-compliant bots can ignore it outright.
Should I block AI crawlers to protect my content?
It is a trade. Blocking training crawlers keeps your words out of future training sets. Blocking retrieval and user-fetch agents also removes you from live AI answers and their citations, which is usually the opposite of what a business wants. Decide per agent, not with one blanket rule.
