GPTBot: what it is and whether to allow it

OpenAI's crawling agent for model training and index refresh. It does not follow links on a user's behalf, and it never clicks anything.

Test whether you are blocking it

curl -sS -A "GPTBot/1.0" -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/

A 403 means a firewall or bot rule matched. A tiny body with status 200 means the crawler reached a shell or a challenge page and read nothing useful. Both are failures; they just look different.

Should you allow it

Appears in logs as GPTBot/1.x. If you want your pages inside OpenAI's index, it must be allowed; blocking it does not stop ChatGPT from citing you through other agents, but it removes one of the two paths in.

Where this fits

Crawler access is check A1 of the fifteen-point audit, and it comes first: nothing downstream matters if the request is refused. The full list is in the standard; the sequence that avoids redoing work is in the repair guide.