Should you let AI crawlers read your website?

Short answer for most small businesses: yes, let the ones that answer questions read you, and make a deliberate choice about the ones that train models. The mistake worth avoiding is blocking all of them by accident — which is easier to do than it sounds, and which some hosting platforms now do by default.

Not all AI crawlers do the same job

“AI bot” covers three quite different things, and the big providers now run them separately so you can treat them differently:

• Training crawlers collect pages that may be used to train future models. OpenAI's GPTBot, Anthropic's ClaudeBot and Common Crawl's CCBot are in this group. Google and Apple use a control token instead — Google-Extended and Applebot-Extended — which you name in robots.txt to opt out of training without affecting normal search crawling.
• Search crawlers build the index an assistant searches when it answers. OAI-SearchBot, Claude-SearchBot and PerplexityBot do this. Block these and you are not in the pool the answer is drawn from.
• User-triggered fetchers visit a page because a person asked an assistant about it, right now. ChatGPT-User, Claude-User and Perplexity-User are examples. Blocking these is the closest thing to hanging up on a customer who asked about you by name.

One trap specific to Google: its AI Overviews are built from the normal Googlebot index. Opting out through Google-Extended does not take you out of AI Overviews; it only affects Gemini training and grounding. The only way out of AI Overviews is out of Google Search, which no small business wants.

The setting that may have changed without you noticing

In July 2025 Cloudflare, which sits in front of a very large share of the web, began offering to block AI crawlers by default when a new domain is added, and followed it with a “managed robots.txt” that writes rules on your behalf. In September 2025 it added a Content Signals line to that file, with three switches: search, ai-input (use in live answers) and ai-train. The default it publishes allows search, refuses training, and leaves AI answers unstated.

None of that is unreasonable. The problem is that it happens at the network layer, not in your website, so your developer's robots.txt can say one thing while the edge serves another. If you have moved a site onto Cloudflare recently — we moved this one in September 2026 — check AI Crawl Control and the bot settings in the dashboard, then open /robots.txt on the live site and read what is actually being served.

How to decide

If you live on being found — a local service, a studio, a shop — allow search crawlers and user fetchers without hesitation. They are how you get named in answers, and an assistant can only cite a page it was allowed to read. We covered what makes a page worth citing in how to get cited by AI search.

Training is a values call, not a traffic one. Blocking training crawlers does not measurably help your rankings, and allowing them does not measurably help either — the effect, if any, is long-term and indirect. Publishers who sell their words have good reason to refuse. A small business whose content exists to be found has less reason, and some reason to want its own description of itself to be the one a model learns.

Remember what robots.txt is. It is a request. The well-known crawlers honour it; a crawler that chooses not to will not be stopped by it. If you have content that genuinely must not be copied, it should not be on a public page.

What about llms.txt?

llms.txt is a 2024 proposal for a plain-text file listing a site's most useful pages, written for language models rather than search engines. It is sensible and cheap. It is also, as of writing, not something any major AI provider has said it relies on, so treat it as a small bet rather than a ranking factor. We publish one, at /llms.txt, because it costs nothing to keep current when it is generated from the same data as the sitemap.

A ten-minute check

1. Open yourdomain/robots.txt on the live site. Read every line.
2. Look for Disallow: / under any of the names above, and for a Content-Signal line you did not write.
3. If you use Cloudflare, check AI Crawl Control and the bot settings, and match them to the decision you actually made.
4. Ask two AI assistants a question your business should answer, and see whether you are cited. Repeat monthly.

Being readable by machines is now part of being findable. If you want a second pair of eyes on it, our SEO service includes this check, and technical SEO that actually moves rankings covers the rest of the foundation.

Sources

• Cloudflare — Managed robots.txt and Content Signals
• Search Engine World — Cloudflare's Content Signals Policy
• Search Engine Land — Anthropic's three crawlers and how to block them