AI crawlers cannot read my Apache site — what to change

On Apache, a page can be readable in a browser and invisible to an AI crawler. Here is what usually causes it, what to change, and what you cannot change from your side.

What usually causes it

.htaccess RewriteRules that refuse unknown user agents, or a security module matching on the AI agent string.

How to confirm it on Apache

curl -sS -A "GPTBot/1.0 (+https://openai.com/gptbot)" -o /tmp/p.html -w "%{http_code} %{size_download}\n" https://example.com/
python3 -c "import re;h=open('/tmp/p.html').read();print(len(re.sub(r'<[^>]+>',' ',h).split()),'words in raw HTML')"

A 403 is a block; a small word count is a rendering failure. They are fixed in different places, so confirm which one you have before changing anything.

What you cannot fix on Apache

Grep your .htaccess files for the agent names, allow them explicitly, and confirm with curl. Re-check after any plugin that rewrites .htaccess on update.

Where this fits

Crawler access is A1 and content in the raw HTML is A2 — both in group A, which runs first because nothing downstream matters if the request is refused. The full definition is in the standard, and the order that avoids redoing work is in the repair guide.