aeolith Check a site
Article

Sitemaps and dead links: the two easy checks nobody runs

Two of the fifteen checks are almost free to pass, and are failed constantly: A3 (a sitemap exists and is submitted) and A4 (no dead-link pollution on core pages). Both are mechanical. Both make everything else work better, because a crawler that trusts your sitemap spends its budget on pages that exist.

Audit the sitemap

curl -sS https://example.com/sitemap.xml | grep -o '<loc>[^<]*</loc>' | sed 's/<[^>]*>//g' > /tmp/urls.txt
wc -l /tmp/urls.txt

while read -r u; do
  code=$(curl -s -o /dev/null -w '%{http_code}' "$u")
  [ "$code" = "200" ] || echo "$code  $u"
done < /tmp/urls.txt

Anything that is not 200 is a problem with two possible fixes:

  • The page should exist → restore it, or remove it from the sitemap.
  • The page is gone on purpose → remove it from the sitemap. A permanent 404 should stay a 404; redirecting it to the homepage teaches crawlers that your redirects mean nothing.

While you are there, check that the sitemap does not list:

  • noindex pages — a page that says "do not index me" inside a file that says "index me" is a contradiction;
  • redirect chains — point at the final URL, not at the first hop;
  • URLs with tracking parameters — they create duplicate pages nobody asked for.
for p in / /pricing /docs; do
  curl -sS "https://example.com$p" | grep -o 'href="[^"#]*"' | sed 's/href="//;s/"$//' \
  | grep -E '^https?://example.com' | sort -u | while read -r u; do
      code=$(curl -s -o /dev/null -w '%{http_code}' "$u")
      [ "$code" = "200" ] || echo "$p → $code $u"
    done
done

Fix in this order: 404s on pages you link from the homepage, then 404s in navigation and the footer, then everything deeper. A dead link in the main navigation is both a user problem and a signal to a crawler that the site is not maintained.

The part that is easy to forget

llms.txt should also contain only live URLs — it is a reading list, and a dead entry makes the whole list untrustworthy. And when you delete a page, remove it from the sitemap, from llms.txt, and from the internal links in the same change. Three files that disagree about which pages exist is the most common source of A3/A4 failures we see.

The report keeps the status code it received for each checked URL, so a failure here is a list of URLs to work through rather than a verdict to argue with.

Check my domain free See all 15 checks
Check a site