Cloudflare AI Crawler Settings Decide Which Assistants Read You

"Blocking AI" is not one setting. Since 15 September 2026, Cloudflare treats it as three: whether AI crawlers can read your site to build a search index, whether they can collect it for training, and whether an agent can open a page because a person asked it to. Those are three different visitors with three different consequences, and your Cloudflare AI crawler settings decide each one separately. If you have never opened them, the defaults decided for you.

The difference matters because only one of the three has much to do with what an assistant says about you today. Training shapes what a model knows months from now. Search and agent fetches shape the answer a buyer gets this afternoon, when ChatGPT or Claude goes and reads your product page. A setting meant to keep your catalogue out of a training set can, if it is the wrong setting, also keep it out of the answer.

What follows is a check you can run in about twenty minutes. It needs a terminal, access to the Cloudflare dashboard for your domain, and your own robots.txt.

What changed at Cloudflare in September 2026

On 15 September 2026 Cloudflare published Have it both ways: stay discoverable in search while disallowing AI training. The old single "Block AI Bots" switch is being deprecated in favour of three controls, Search, Training and Agent. Each takes one of four values: Allow, Block, Block on pages with ads, and, for Training only, Disallow AI Training.

Disallow AI Training is the new middle option. Googlebot, Bingbot and Applebot, which crawl for both search and training, stay allowed for search. Training-only crawlers, including those of Amazon, Anthropic, Meta and OpenAI, are blocked. The flip side is less obvious: Block and Block on pages with ads now apply to those mixed crawlers too, which means they also stop them for search.

The same post deprecates "Managed robots.txt" in favour of Bot Preference Sync, announced on 21 August 2026 and updated on 15 September. It writes your choices into robots.txt for you, and it is on by default for new customers.

Step 1. Read the robots.txt that is actually served

The file in your repository or CMS is not necessarily the file crawlers receive. Bot Preference Sync prepends its own block to whatever is already there, so the served file can be longer than the one you wrote. Fetch it from outside:

curl -s https://www.example.com/robots.txt

Look for a section that starts with # BEGIN Cloudflare Bot Preference Sync and ends with # END Cloudflare Bot Preference Sync. Everything between those two lines was written by the CDN, from your dashboard settings, from a bot list Cloudflare maintains and updates. Write down which user agents appear there with Disallow: /. If the markers are missing, either the sync is off or your site is not behind Cloudflare, and the rest of this protocol applies only partly.

Step 2. Write down the three values in the dashboard

Open the AI crawler controls for the zone and note the value of Search, Training and Agent. Three words on paper are enough. Then compare them with the defaults Cloudflare documents for domains added from 15 September:

• a domain that does not monetise with ads: Allow, Allow, Allow;
• a domain that does: Search Allow, Training Disallow AI Training, Agent Block on pages with ads.

Older domains were migrated rather than reset. A domain that had the legacy "Block AI Bots" on, in either form, was mapped to the second set. So a site that switched on the old block years ago may now carry an Agent restriction nobody chose on purpose.

Step 3. Map each AI crawler to a Cloudflare control

Each assistant uses several crawlers, and Cloudflare files them in different categories. This is how its public bot directory classified them when we checked on 20 October 2026:

• OpenAI: GPTBot is Training, OAI-SearchBot is Search, ChatGPT-User is Agent.
• Anthropic: ClaudeBot is Training, Claude-SearchBot is Search, Claude-User is Agent.
• Google, Microsoft, Apple: Googlebot, Bingbot and Applebot are mixed, search and training at once.

So Disallow AI Training touches GPTBot and ClaudeBot and leaves the other four alone. OAI-SearchBot and Claude-SearchBot answer to the Search control. ChatGPT-User and Claude-User answer to the Agent control. These last two are the fetches that happen while a person is waiting for an answer: someone pastes your URL into ChatGPT, or Claude decides to open your pricing page. Cloudflare describes their access as "intermediary", meaning an end user drives them.

Perplexity is not in that directory. Cloudflare removed it from its verified bots on 4 August 2025, after reporting undeclared crawling. The posts of September 2026 do not say how the three controls treat Perplexity's crawlers, so we will not guess. If Perplexity matters to your market, it belongs in step 5, the test from outside.

Step 4. Check the Agent value against your pages

This is the step that catches most sites by surprise. With Agent set to Block on pages with ads, ChatGPT-User and Claude-User are refused on every page that Cloudflare detects as serving ads. Training is untouched, search indexing is untouched, and the one moment when an assistant tries to read your page for a live question is the one that fails.

For a B2B manufacturer with no advertising, the value is probably Allow and the step takes ten seconds. It deserves more care if the site runs ads anywhere: a blog monetised with a display network, a trade directory section, a page with sponsored placements. Which pages count is Cloudflare's detection, not your list, so check the pages that matter commercially, product, pricing, distributor pages, rather than assuming.

The same reasoning applies to Training. If its value is Block rather than Disallow AI Training, Googlebot, Bingbot and Applebot are stopped for search as well. That is a legitimate choice for some publishers. For a company that wants to be found, it is rarely the intended one.

Step 5. Test from outside, with real assistants

A spoofed user agent from your laptop proves nothing: Cloudflare verifies bots by more than the name they send, and your request is not the bot. The honest test uses the assistants themselves.

1. Pick three URLs: the home page, a product page, a page with ads if you have one.
2. In ChatGPT and in Claude, paste each URL and ask for a two line summary of what is on the page.
3. In your logs or in Cloudflare's security events, filter for ChatGPT-User and Claude-User over the same minutes, and note the status code: 200 means served, 403 means refused.
4. Repeat in Perplexity and look at what reaches you, if anything.

Six requests, two minutes each. If an assistant answers with something vague about not being able to access the page, and your log shows a 403 at that minute, you have found the setting. Our guide to what an AI crawler looks like in your server logs shows how to read those lines.

Step 6. Record what you decided, and when

Write the three values, the date and the reason in the same place you keep your robots.txt notes. Two dates are worth adding to the calendar. Microsoft plans to honour a "no training" preference in robots.txt from early 2027; until then, Disallow AI Training sends no robots.txt signal to Bing. And Cloudflare updates its bot list over time, so the block in step 1 can change without anyone touching the dashboard.

There is a limit to what this protocol tells you. It establishes whether assistants can read your pages, not whether they mention you. Access is a precondition, never a guarantee. Measuring the answers themselves, in each language and market, is what PSentry does, twice a month across ChatGPT, Claude, Gemini and Perplexity. If a change in the scores lines up with the date you changed a CDN setting, the note from this step is how you will notice. For the older half of the robots.txt story, what llms.txt does and does not do, see AI crawlers and llms.txt.