Ai Search

Should I block AI crawlers on my site?

For almost any business that wants to be found, the answer is no. The crawler you would block is usually the very one that fetches your page to quote and cite it in an AI answer, so blocking it does not shield you from AI. It deletes you from the results.

The real decision is not a simple on or off switch. Two different kinds of AI crawler do two different jobs, and a well-built robots.txt treats them differently. Confusing the two is how businesses quietly opt out of the exact traffic they were trying to win.

Two kinds of AI crawler, two different jobs

Every major AI platform runs more than one bot, and lumping them together is the mistake that costs you visibility. Before you touch a single line of robots.txt, you have to know which bot you are actually talking to.

Training crawlers

GPTBot from OpenAI, Google-Extended, ClaudeBot, and CCBot from Common Crawl collect pages to train or ground future models. Blocking them opts you out of being learned from. That does not remove you from a live answer, and on its own it does not win you a citation either way. It is a data-rights decision, not a visibility one.

Retrieval crawlers

OAI-SearchBot and ChatGPT-User from OpenAI, PerplexityBot and Perplexity-User, and plain Googlebot fetch pages to build the search index an assistant reads from, or to open your page in real time while it answers a prompt. These are the bots that produce a citation with a link back to you. Block these and you cannot be cited at all.

The retrieval path is worth picturing. A user asks an assistant a question. The assistant checks its search index, shortlists a few promising sources, and in many cases fetches those pages live to pull exact wording. Your page can only enter that shortlist if its crawler was allowed in. Block the crawler and you are not a slow result, you are simply not a candidate.

What blocking actually does, and what it does not

A robots.txt entry is a public request, not a lock. Well-behaved crawlers honor it. A scraper that wants your content ignores it. So if a bot is hammering your server, the real fix is server-level, a firewall rule or a rate limit, not a line in a file that bad actors skip.

It also helps to separate a citation from a mention. A citation is a visible link to your page as the source of a claim. A mention is the model naming your business from memory, with no link. Blocking training crawlers slowly starves future mentions. Blocking retrieval crawlers kills citations today. Most businesses want both kinds of exposure, and very few genuinely want neither.

Googlebot is the common trap. Google's AI Overviews are built from the ordinary Search index, so if you block Googlebot to stay out of AI, you also fall out of standard Google results. The lever for Gemini training is Google-Extended, and it does not touch Search at all.

When blocking is the right call

There are legitimate reasons to disallow a crawler. They are narrower than most people assume.

Staging, admin, and other URLs that should never be public to any visitor.

Proprietary or paywalled content you sell, where being quoted for free undercuts the product.

A single misbehaving bot that ignores crawl-rate etiquette, which you throttle at the server, not only in robots.txt.

A deliberate, documented choice to stay out of model training while still allowing retrieval, set per user agent.

Notice what is not on that list: your marketing pages, service descriptions, service areas, and FAQs. Anything you would happily hand a prospect is content you want crawled, indexed, and quoted. Hiding it from AI does not protect it, it just makes a competitor the easier source to cite.

How to decide for your own site

Work it through in this order rather than copying someone else's robots.txt.

Decide what AI visibility is worth to you. If buyers could find you through an assistant, retrieval access is not optional.

Separate public marketing from private and proprietary URLs. Only the second group is a real blocking candidate.

Treat training as a separate question from retrieval. You can decline one and keep the other.

Confirm Googlebot stays open so you keep both standard Search and AI Overviews.

Check your robots.txt against a current list of crawler user agents, because platforms add and rename bots often.

The setup most businesses actually want

For a site whose job is to win customers, the sensible default is to allow retrieval crawlers everywhere, make a deliberate per-platform decision on training crawlers, and keep Googlebot fully open. That is a handful of lines in one file. Getting the user agents and the logic right is where it usually goes wrong.

When we handle AI search optimization, crawler access is one of the deliverables, not an afterthought. We read what your current robots.txt is actually telling each bot, set an allowlist that keeps you eligible for citations, document every decision against an up-to-date crawler reference, and monitor it so a later change does not silently lock the assistants out. We do this only on sites we build and maintain, under a published monthly retainer with a 90 day initial term. You can see how that is structured at /pricing/seo/ and what the wider service covers at /services/seo/.

Questions

Will blocking GPTBot remove my site from ChatGPT?

Not reliably. GPTBot is OpenAI's training crawler, so blocking it opts you out of future model training, but ChatGPT can still reach you in live answers through OAI-SearchBot and ChatGPT-User. If your goal is to stay out of answers entirely you would have to block those too, and then you also lose every citation and the referral clicks that come with them.

Is robots.txt enough to stop AI scraping?

No. A robots.txt entry is a public request that only well-behaved crawlers honor. A scraper that wants your content can read the file and ignore it. To actually stop a bot you have to block it at the server with a firewall rule or a rate limit.

Does blocking AI crawlers help my SEO or hurt it?

Blocking the retrieval crawlers hurts you, because those are the bots that let you be cited in AI answers. Blocking training-only crawlers has no direct effect on your Google rankings. Never block Googlebot to avoid AI, because AI Overviews are built from the same Search index as your regular results.

Can I block AI training but still show up in AI answers?

Yes, and for many businesses that is the balanced choice. You disallow the training user agents like GPTBot, Google-Extended, and CCBot, while allowing the retrieval user agents like OAI-SearchBot, PerplexityBot, and Googlebot. The decision is made per bot in robots.txt, which is why the user agent names have to be exactly right.

Should a local service business ever block AI crawlers?

Rarely. A local service business lives on being found, and assistants are becoming a common first stop for buyers comparing options. Blocking the crawlers that read your pages takes you out of that conversation. The only pages worth protecting are staging and private URLs, not your public marketing.