AI crawlers cannot read my Nginx site — what to change

On Nginx, a page can be readable in a browser and invisible to an AI crawler. Here is what usually causes it, what to change, and what you cannot change from your side.

What usually causes it

An if ($http_user_agent ~* ...) block, a map returning 403 for AI agents, or a rate limit that trips on crawl bursts. Sometimes it is a challenge page returning 200 with no content.

How to confirm it on Nginx

curl -sS -A "GPTBot/1.0 (+https://openai.com/gptbot)" -o /tmp/p.html -w "%{http_code} %{size_download}\n" https://example.com/
python3 -c "import re;h=open('/tmp/p.html').read();print(len(re.sub(r'<[^>]+>',' ',h).split()),'words in raw HTML')"

A 403 is a block; a small word count is a rendering failure. They are fixed in different places, so confirm which one you have before changing anything.

What you cannot fix on Nginx

Search the site config for the AI user agents, then remove the block or return 200 for them; verify per agent with curl and check the access log for the exact status the agent received.

Where this fits

Crawler access is A1 and content in the raw HTML is A2 — both in group A, which runs first because nothing downstream matters if the request is refused. The full definition is in the standard, and the order that avoids redoing work is in the repair guide.