Crawler access for AI user agents

Crawler access for AI user agents is check A1 in the fifteen-point audit. The AI user agents we test (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) are not blocked by robots.txt, a firewall rule or a bot challenge.

What counts as evidence

The HTTP status and byte size we received for each agent, next to the same request made with a normal desktop user agent.

How to fix it

Allowlist the AI agents by user agent and path, and keep the JS challenge off for crawlers. Re-test with the same curl commands.

Where this sits

A is the group: accessibility, comprehension, or extractability. The groups run in order — a crawler that cannot reach the page never reaches the reading checks — so fix the earliest failing group first. The full definition of all fifteen items is in the standard.