OAI-SearchBot: what it is and whether to allow it
OpenAI's search index crawler, separate from GPTBot and from ChatGPT-User. It is the agent that decides what search results inside ChatGPT can cite.
Test whether you are blocking it
curl -sS -A "OAI-SearchBot/1.0" -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/
A 403 means a firewall or bot rule matched. A tiny body with status 200 means the crawler
reached a shell or a challenge page and read nothing useful. Both are failures; they just look
different.
Should you allow it
A common failure: an operator blocks GPTBot for training reasons and accidentally blocks OAI-SearchBot too, which silently removes them from AI search results while the site looks perfectly healthy in a browser.
Where this fits
Crawler access is check A1 of the fifteen-point audit, and it comes first: nothing downstream matters if the request is refused. The full list is in the standard; the sequence that avoids redoing work is in the repair guide.