How to tell whether AI assistants are actually sending you traffic

Two different things get called "AI traffic", and mixing them up leads to wrong conclusions:

• Crawling: an AI company's crawler fetches your pages to build or refresh an index. It appears in your logs as a bot user agent, and it never clicks anything.
• Referral: a person asked an assistant a question, the assistant cited you, and the person followed the link. It appears as a normal browser hit with a Referer header or an utm_source you can recognise.

A log line proves crawling. It does not prove that anyone read the answer.

What to grep for

# 抓取：这些是爬虫，不是访客
grep -iE "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended" \
     /www/wwwlogs/example.com.log | awk '{print $1, $12, $7}' | sort | uniq -c | sort -rn | head

# 引荐：真人点击（Referer 里出现 AI 域名，或登录页带的参数）
grep -iE "referer.*(chatgpt\.com|perplexity\.ai|claude\.ai|gemini\.google\.com)" \
     /www/wwwlogs/example.com.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

ChatGPT-User is the one that follows a link on a user's behalf, so it is the closest thing to a referral on the crawling side. Treat it as a strong hint, not as proof of a human.

Why a hit is not a referral

A bot fetch tells you the URL is reachable and parseable. It says nothing about whether the content answered a question. The signal that matters is whether the fetched page contains a quotable answer: a conclusion near the top, a specific sentence, and evidence for it.

That is exactly what checks C3 (extractable structure) and C5 (quotable conclusion) measure. If those fail, the crawl can be perfectly healthy and still produce no citations.

A ten-minute routine

1. Count crawler hits per URL for the last week. Pages fetched repeatedly are the ones the assistant considers authoritative — make sure they are the right pages.
2. Check the 200 rate for those fetches. A page that answers 403 to GPTBot is a page that cannot be cited, however good it is.
3. Compare the fetched URLs against your sitemap. Pages nobody fetches are usually pages nothing links to.
4. Only then look at referrals. If citations are missing while crawling is healthy, the problem is in the content, not in the delivery.

The free report covers the delivery side — access, raw HTML, structure — and records the exact response we received, so the two halves of the question stay separate.