CCBot: what it is and whether to allow it

Common Crawl's crawler. Its dumps are the raw material many models are trained on, which is why it is the agent most often blocked outright.

Test whether you are blocking it

curl -sS -A "CCBot/1.0" -o /dev/null -w "%{http_code} %{size_download}\n" https://example.com/

A 403 means a firewall or bot rule matched. A tiny body with status 200 means the crawler reached a shell or a challenge page and read nothing useful. Both are failures; they just look different.

Should you allow it

Blocking CCBot has no effect on Google, Bing or the assistants' own crawlers. It is a training-data decision, not a visibility one.

Where this fits

Crawler access is check A1 of the fifteen-point audit, and it comes first: nothing downstream matters if the request is refused. The full list is in the standard; the sequence that avoids redoing work is in the repair guide.