AI Crawlers and llms.txt: What Is Really Blocking the Bots

Before you read the rest of this, do the check. Open a new tab, type your domain followed by /robots.txt, and read what comes back. It takes ten seconds, and it is one of the few parts of AI visibility that gives a definite answer.

You are looking for lines like User-agent: GPTBot followed by Disallow: /. Also ClaudeBot, PerplexityBot, Google-Extended. If they are there, someone on your side put them there, and there is a decent chance nobody in the room today knows who or why. These rules spread during the scraping panic: in template files, in hardening guides, in plugins nobody has opened since. A rule added once to keep the machines out is still keeping them out, and the machines are now where a portion of your buyers ask their first question about your category.

The harder case is the site whose robots.txt permits everything and whose bots are still turned away at the door, by a CDN that never consults it at all.

The distinction that changes the decision

Almost every discussion of this treats AI crawlers as one thing to allow or deny. They fall into two categories, and blocking is defensible for one and self-defeating for the other.

Crawlers that collect training data

These fetch pages in bulk, on their own schedule, and the text ends up in the corpus a future model is trained on. GPTBot is OpenAI's. ClaudeBot is Anthropic's. Nobody is standing at the other end waiting for the page. This is the crawler people mean when they talk about AI companies taking content without paying for it, and blocking it is a legitimate position on the ownership of your writing. A publisher with an archive worth licensing has a real argument for the block. So do you, if that is what you believe.

What you give up is diffuse, and worth less than most AI visibility advice implies: a model trained on the whole web has read about you from sources you never controlled anyway.

Fetchers that retrieve a page in real time

Here is where the damage happens quietly. When the assistant decides a question needs current information, a different agent fetches pages, right then, to write the answer in front of the person. OpenAI sends OAI-SearchBot and ChatGPT-User. Anthropic uses separate user agents for user-initiated fetches. Perplexity runs its own retrieval, which is most of what Perplexity is.

Blocking these protects your copyright in no meaningful sense. The content is not going into a training set. It is being read to answer a question a person is asking right now, possibly about you. Deny that request and the assistant reaches for whoever did allow it, and writes their name instead of yours. You have not defended anything. You have removed yourself from the sentence.

Two separate decisions, so make them separately. Blocking training while allowing retrieval is coherent, and probably the right default for a company that sells something rather than publishes something. What you should not do is block both because a line in a template did it for you.

The one that is not a crawler at all

Google-Extended fetches nothing. It is a control token: Googlebot crawls your site as it always did, and Google-Extended governs whether that already-crawled content may be used for Gemini. Disallowing it does not remove you from search. Allowing it adds no new crawler to your logs. It is a permission expressed in a file designed for something else, which is why people reason about it wrongly.

The blocks you cannot see from a browser

A company checks robots.txt, finds it permissive, and moves on. Meanwhile the requests never reach the application. Bot management, rate limiting and application firewalls work on user agent strings, on the reputation of the requesting network, on behavioural signatures. Most ship with rules aimed at scrapers, and AI crawlers look exactly like scrapers, because in a narrow technical sense they are. The request is refused, or fed a challenge it cannot solve, and none of that shows up in a file you can inspect. Your robots.txt is a polite notice on the door. Your CDN decides who gets in, and the two are not talking.

Rendering is the other silent one. Retrieval agents are typically not full browsers. If a page composes its main content with JavaScript, whatever fetches it may receive an empty shell, and words assembled after a script runs might as well not be on the page.

llms.txt, honestly

llms.txt is a proposed convention: a plain markdown file at your root offering a clean, ordered map of your important content, so a model does not have to fight through navigation and layout to find the substance. The idea is sound, and writing one forces you to state plainly what your product is.

Now the honest part, because this file is being sold badly. It is a proposal, not a standard. Nobody has agreed to obey it, no major assistant has committed to reading it, and support may never arrive from the vendors that matter to you. It does not grant permission, it does not remove a block, and it obliges nobody to cite you. A model that ignores llms.txt is not misbehaving, because there is no rule to break. Publishing one costs an afternoon and might help. Treating it as the key that unlocks AI visibility is a claim nobody can support.

A verification protocol you can run this week

All of it is falsifiable: you either find the problem or you rule it out.

Read the file. Search yourdomain.com/robots.txt for each agent by name: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, plus the user-initiated fetchers each vendor documents. Then find out who added the block, and whether it was a copyright decision or a copied snippet.

Request your own pages as the bots do. From a terminal, fetch a key page while sending the user agent string of each AI crawler, and compare it with what an ordinary browser gets. If the browser receives the page and the bot receives a refusal, a challenge, or an empty body, the block is not in robots.txt. It is at your edge. This single command catches the most common invisible failure here.

Load your pages without JavaScript. Open your product page, your pricing page and your documentation with scripting disabled. If the content that explains what you sell is missing, a retrieval agent may see the same emptiness.

Look in the logs. Ground truth, and almost nobody does it. Grep your access logs for the AI user agents. Are they arriving, on which paths, and are they getting the page or a refusal? A site that allows everything and still sees no AI crawler traffic has a different problem from a site that blocks them.

Re-check after any infrastructure change. A new CDN, a security plugin, a migration. Bot rules get reintroduced by default configurations, not by decisions.

What this gets you, and what it does not

Unblocking the crawlers is necessary and it is not sufficient. It makes you retrievable. It does not make you recommended. A model that can reach your page still has to decide the page is worth reaching for, and that rests on what the whole web says about you, in each language, over years. Access is the floor of the building, not the building.

Which is why the check has to be paired with measurement: fixing a block only means something if you know whether the assistants named you before, and whether they name you after. That is what PSentry does. It runs your prompt set across ChatGPT, Claude, Gemini and Perplexity, in each language you sell in, and reports where you were named, where you were cited, and who was recommended in your place. What it will not do is take a side in what a model decides to say. Reading outcomes and authoring them are different jobs, and this tool only ever does the first: opening a crawler is not a purchase that guarantees a better answer next time you check. You get the honest comparison, before the change and after, and nothing shaped in between.

Frequently Asked Questions

If I block GPTBot, does my brand disappear from ChatGPT?

No. Blocking a training crawler does not erase what a model already absorbed, and it does not stop the assistant describing you from press, forums and reviews. What it removes is your own site as a direct future input. Blocking the live retrieval agents is the more immediate loss, because those fetch pages while a user waits.

Is llms.txt actually read by ChatGPT, Claude, Gemini or Perplexity?

There is no vendor commitment that says so, and you should not plan around one. Treat it as a low-cost bet with an uncertain payoff. Anyone telling you the major assistants consume it today is claiming more than is knowable.

My robots.txt allows everything. Why do I see no AI bots in my logs?

Most likely something in front of your application is refusing them: a CDN bot rule, a firewall signature, an aggressive rate limit. Fetch your own page while presenting an AI user agent and compare it with a normal browser request. If they differ, you have found it. The other explanation is that nothing on your site is worth retrieving yet, which is a different problem.

How quickly does unblocking show up in AI answers?

Slowly, and unevenly. Live retrieval can pick you up as soon as the block is gone. Anything that depends on training absorbs your content on a cycle you neither control nor observe. This is a change you measure over months, which is also why alerting on it in real time would tell you nothing.