Which AI crawlers should a practice allow, and which can it block?

A practice asked us why it had vanished from every assistant at once. Nothing had changed on the website. Their host had turned on a new bot-protection setting three weeks earlier, and every AI crawler was getting a 403 before it reached a single page. robots.txt was immaculate, and irrelevant.

The two kinds of crawler, in the vendors’ own words

Retrieval crawlers fetch pages so an assistant can answer with them now. Training crawlers collect content that may go into a future model. OpenAI, Anthropic and Perplexity each document which of their agents does which, and the names are close enough that people block the wrong one by accident.

What a working robots.txt looks like

Simple is safer than clever. A wildcard that allows everything except your admin and API paths already permits every crawler in the table. Name the retrieval agents explicitly only if you also intend to restrict the wildcard later, and check the file at your real domain after any site change.

1. Open https://yourpractice.com/robots.txt in a browser and read what is actually served. Many sites serve a file nobody on the team has ever seen.
2. Confirm nothing disallows the whole site. A stray "Disallow: /" left over from a staging setup is the most common cause of total invisibility.
3. Ask your host or developer to check the CDN or firewall rules for these agent names specifically, and to look at request logs for them.
4. Decide the training question deliberately, write the decision down, and revisit it once a year along with the crawler names, which change.

Should you let the training crawlers in?

There is a real trade here and no consensus. Allowing them means your material may inform how future models describe your field, and your practice, without attribution or a link. Blocking them protects your content and costs nothing in today’s answers, because retrieval is a separate path with separate agents.

Our default for clients is to allow retrieval, block training, and revisit. If you publish original research or a distinctive method you would rather not see absorbed, blocking is reasonable. If your material is ordinary service copy, blocking gains you little.

The limits of robots.txt

It is a request, not a wall. Well-behaved crawlers respect it and others do not, and Perplexity documents that a user-initiated fetch behaves differently from indexing. If content genuinely must not be read by a machine, it belongs behind a login, not behind a line in a text file.