AI crawlers cannot read my Cloudflare site — what to change

On Cloudflare, a page can be readable in a browser and invisible to an AI crawler. Here is what usually causes it, what to change, and what you cannot change from your side.

What usually causes it

Bot Fight Mode, a managed WAF rule or a "block AI bots" toggle refuses the agent before the request reaches your origin. Your application logs show nothing because the request never arrived.

How to confirm it on Cloudflare

curl -sS -A "GPTBot/1.0 (+https://openai.com/gptbot)" -o /tmp/p.html -w "%{http_code} %{size_download}\n" https://example.com/
python3 -c "import re;h=open('/tmp/p.html').read();print(len(re.sub(r'<[^>]+>',' ',h).split()),'words in raw HTML')"

A 403 is a block; a small word count is a rendering failure. They are fixed in different places, so confirm which one you have before changing anything.

What you cannot fix on Cloudflare

Add a WAF skip rule for the verified AI agents (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot), and check Cloudflare's own analytics rather than your origin logs — that is the only place the block is visible.

Where this fits

Crawler access is A1 and content in the raw HTML is A2 — both in group A, which runs first because nothing downstream matters if the request is refused. The full definition is in the standard, and the order that avoids redoing work is in the repair guide.