aeolith Check a site
Article

Why a WAF or CDN rule can hide your site from GPTBot

A site can look perfect in a browser and be completely unreadable to an AI crawler. The usual cause is not your markup — it is a firewall, a CDN rule, or a robots.txt line that was written to stop scrapers and now stops GPTBot and ClaudeBot too. This is check A1 (crawler access) in our 15-point audit, and it fails silently: nothing errors, the site simply never appears.

How to tell a block from an unreadable page

Ask for the page the way a crawler does — no cookies, no JavaScript, an AI user agent:

curl -sS -A "GPTBot/1.0 (+https://openai.com/gptbot)" -o /tmp/page.html \
     -w "%{http_code} %{size_download}\n" https://example.com/

curl -sS -A "ClaudeBot/1.0" -o /dev/null -w "%{http_code}\n" https://example.com/

Three outcomes, three different problems:

What you see What it means Where to fix it
403 or 406 A WAF/CDN rule matched the user agent Firewall or bot rules
200 with a tiny body JavaScript-only shell (check A2) Rendering strategy
200 with real text Access is fine Look at the reading checks

If the 403 only appears for AI user agents and not for your browser, you have found the cause. Compare with a normal desktop agent in the same command to be sure it is the user agent and not a rate limit kicking in at that moment.

Check robots.txt for the names, not the intent

Many robots.txt files block GPTBot because it was added by a plugin during a "block AI training" week, and nobody revisited it:

curl -sS https://example.com/robots.txt | grep -i -E "gptbot|claudebot|perplexity|ccbot|google-extended"

Disallow: / for those agents is a decision, not a bug — but it should be a decision you can name. If the goal is only to stop training crawlers while staying readable for retrieval, say so explicitly, and do not leave the whole site blocked by accident.

Fix the block without opening the door

  1. Allowlist by user agent and path, not by IP: AI crawlers do not publish stable IP ranges you can pin.
  2. Keep the bot challenge off for crawlers: a JavaScript challenge page returns 200 with the challenge body — an AI crawler reads the challenge and nothing else.
  3. Serve the same HTML to crawlers and to people. Cloaking is both detectable and unnecessary: the content is the same, only the rendering budget differs.
  4. Re-check after the change with the same curl commands. A fix you did not re-measure is a hope.

What the report shows

Every report records the response we actually received for each checked URL — status code, byte size and the moment of the fetch. That is what makes a blocked crawl distinguishable from an unreadable page: the report says 403 instead of guessing, and the repair note points at the infrastructure layer rather than at your content.

If you have not run the check yet, the free report at the checker covers A1 and the other fourteen items and keeps the evidence beside each verdict.

Check my domain free See all 15 checks
Check a site