Category
AI crawlers
What each AI crawler does, whether you are blocking it, and which of the two decisions (training vs retrieval) you are actually making.
Pages
-
CCBot: what it is and whether to allow it
-
Google-Extended: what it is and whether to allow it
-
PerplexityBot: what it is and whether to allow it
-
ClaudeBot: what it is and whether to allow it
-
ChatGPT-User: what it is and whether to allow it
-
OAI-SearchBot: what it is and whether to allow it
-
GPTBot: what it is and whether to allow it