Ai Search

AI crawler access: which bots to let in, and what it buys you

An AI crawler is an automated client that fetches your web pages on behalf of a language model, either to help train it or to answer someone's live question with your content attached. Whether those crawlers are allowed through your site, and whether what they find is worth quoting, is the difference between turning up in an AI answer and being invisible to one.

That makes AI crawler access a build decision before it is a marketing one. You cannot be cited by a system that was never allowed to read you, and you cannot be cited well by a page that loads slowly, hides its answer, or contradicts its own structured data. This page covers both halves: which crawlers matter, and what letting them in actually buys you.

The two kinds of AI crawler

Lumping every bot together is the first mistake. There are two jobs an AI crawler does, and they are controlled separately.

Training crawlers

These collect pages to build or refine a model. They run on their own schedule, months before anyone asks a question, and what they take becomes part of the model's general knowledge rather than a live citation. Blocking them is a reasonable choice about how your content is used, and it does not, on its own, remove you from live answers.

Retrieval and live-fetch crawlers

These fetch a page at the moment of a question, read it, and may quote it in the answer that comes back seconds later. This is the traffic that puts you in a cited answer. Block it and you are not in that answer at all, no matter how good the page is.

Where access is actually decided

A business owner usually assumes the site is open to everyone. In practice, access gets refused in places nobody looks. The first is robots.txt, where a single line naming a user agent either lets it in or shuts it out. The second is your CDN or firewall, which may be fighting bots aggressively and treating a legitimate AI crawler as an attacker. The third is rendering: if your content only appears after JavaScript runs, a crawler that does not execute it sees an empty page.

None of these are visible from the front end. A site can look perfect in a browser and be closed, slow, or blank to half the crawlers that matter. The only honest way to know is to read the server logs and see which agents came, what status code they got, and what they were served.

A crawler that hits a challenge page, a rate limit, or a redirect loop logs a failure, not a read, and a page that is never read is never quoted. These failures are routine and almost always unintentional, which is why nobody catches them without opening the logs.

Allowing a crawler is not the same as getting cited

Access is necessary. It is not sufficient. Letting a crawler in gets your page read. It does not make your page the thing worth quoting, and the gap between those two is where most of the work lives.

A mention is when a model repeats something about you from memory. A citation is when it fetches a specific page, quotes it, and links it as the source. You want the citation, because it is current, attributable, and sends a reader to you. Earning it means answering the question in the first paragraph, making one claim per sentence so a line can be lifted without becoming false, and keeping the structured data honest about what the page says.

This is also why a page can be mentioned and never cited. The model remembers a stale fact about you but has no fresh page it is willing to quote, so the answer talks about your business without sending anyone to it. Fixing that is less about volume of content and more about making one clean, current page that a crawler can reach and a model can trust.

Should you block AI crawlers?

Sometimes yes. If you are protecting original work from training use, blocking the training crawlers is defensible, and you can do it while leaving the retrieval crawlers open so you still appear in live answers. What rarely makes sense is the accidental block: a blanket CDN rule or a copied robots.txt that quietly shuts out the very bots that would have cited you. We treat that as a bug, because it is one.

The decision is per crawler, not all or nothing, and it should be made on purpose. We walk through the tradeoff in plain terms before touching a single line.

How we handle AI crawler access

We do SEO on websites we build. That is deliberate here, because every lever that controls AI crawler access is a build decision: the robots.txt file, the CDN rules, how the page renders, how fast the first screen arrives, and whether the structured data agrees with the visible text. On a site we did not build, we would be guessing at all of it.

The work is concrete. We author robots.txt per crawler rather than copying a default, so training and retrieval are decided separately and on purpose. We check that the CDN is not silently blocking legitimate agents. We make sure the answer is in the HTML, not stranded behind a script. Then we read the server logs on a schedule to confirm which crawlers actually came and what they took, and we tell you plainly when a change made no difference. It sits inside the search retainer rather than being sold as a separate line. You can see how the model is priced on the pricing page.

What we will not do is promise you a citation. Nobody controls which sources a model selects, and that selection shifts between model versions. We make the page the obvious thing to quote, open the door to the crawlers that do the quoting, and then measure whether it worked.