Check
ClaudeBot: what it is and whether to allow it
Anthropic's crawler. Claude also fetches pages for a user's request through a separate agent; ClaudeBot itself is the indexing side.
Test whether you are blocking it
curl -sS -A "ClaudeBot/1.0" -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/
A 403 means a firewall or bot rule matched. A tiny body with status 200 means the crawler
reached a shell or a challenge page and read nothing useful. Both are failures; they just look
different.
Should you allow it
Check both robots.txt and any WAF rule: the most common block is a generic "block AI crawlers" rule that lists ClaudeBot along with scrapers.
Where this fits
Crawler access is check A1 of the fifteen-point audit, and it comes first: nothing downstream matters if the request is refused. The full list is in the standard; the sequence that avoids redoing work is in the repair guide.