robots.txt lets the AI crawlers in
Passes when the user agents used by the assistant crawlers are not disallowed, for the paths we fetch. Commonly fails because a broad Disallow: / written for scrapers also covers every AI agent.
Features
What the check actually does
Fifteen items, three questions: can a crawler reach you, can it understand you, and would an assistant quote you. Below is what each item measures, what a pass requires, and how it usually fails.
Every rule here is published in the standard; the way a measurement is taken is in the methodology.
Group A · Reachability
Can an AI crawler get in and read the page at all?
5 checks in this group.
Content is in the raw HTML
Passes when the text is present in the response body without executing scripts. Commonly fails on client-rendered apps, where the HTML is a shell and the words arrive later.
A sitemap exists and is submitted
Passes when a valid sitemap is served and referenced from robots.txt. Commonly fails because it exists but lists URLs that redirect, or was never updated after a migration.
No dead-link pollution on core pages
Passes when the pages we can reach from the site's own navigation all return 200. Commonly fails after a redesign leaves footer links pointing at removed pages.
The site is stable and HTTPS is valid
Passes when the pages respond consistently and the certificate chain validates for the host. Commonly fails when the apex and the www host disagree about redirects or certificates.
Group B · Comprehension
Can a machine work out who you are and what the page says?
5 checks in this group.
Unique title and description per page
Passes when every checked page has its own title and description. Commonly fails when a template falls back to the site name on hundreds of pages at once.
Structured data is present and valid
Passes when the JSON-LD parses and describes the entity the page claims. Commonly fails when a plugin emits the block with an outdated type, or emits nothing on article pages.
Semantic heading hierarchy
Passes when there is one h1 per page and headings descend without skipping. Commonly fails when a theme uses h3 for visual size rather than structure.
An llms.txt tells AI what matters
Passes when the file exists and follows the format: a heading, a summary, resolvable links. Commonly fails because a placeholder file was uploaded and never filled in.
The operator behind the site is verifiable
Passes when a real organisation, address or contact route is stated in the markup and on the page. Commonly fails on sites with no About page and no structured organisation data.
Group C · Citation readiness
Would an assistant choose to quote this page?
5 checks in this group.
The conclusion comes first
Passes when the main answer appears in the opening block rather than at the end. Commonly fails on narrative pages that build up to the point.
Extractable structure: lists, tables, steps
Passes when the page carries machine-extractable structure rather than only prose. Commonly fails on pages that present a procedure as six paragraphs.
Original data or first-hand information
Passes when the page contains something that exists only here — a measurement, a dataset, a published method. Commonly fails on rewritten summaries of other people's pages.
A visible last-updated date
Passes when a date is visible to a reader and present in the markup. Commonly fails because the only date on the page is a copyright year in the footer.
Third-party references and mentions
Passes when other sites reference or cite you in a way we can observe. Commonly fails for new domains, and is the slowest item to move — which is why it is reported last.
Deliverables
What you get back
Three artefacts, all readable without an account, all pointing at evidence rather than at a score.
Report
The 15-point report
Every check with its raw evidence and the date we saw it: the response we received, the markup we parsed, and the rule a pass requires. Plain HTML, no JavaScript needed to read it.
Fix bundle
The repair pack
The specific change that turns each failure into a pass, per platform, written to be handed to whoever ships the site rather than to be read for inspiration.
History
The re-check history
Run the same check after you deploy. Same fifteen items, side by side, so improvement is a recorded fact instead of a claim.
Scope
How this differs from the tools you already have
Different questions, different answers. None of the three replaces the others.
Crawl and rank
SEO audit
Optimises for a ranked list of links: keywords, internal linking, Core Web Vitals. Useful, and orthogonal — a page can rank and still be unreadable to a machine that will not run your JavaScript.
Markup syntax
Schema validator
Checks whether JSON-LD parses and matches a schema type. This check asks a further question: does the entity it describes match the page, and is it the entity an assistant needs.
Mentions over time
AI monitoring
Tracks whether assistants mention your brand. That is an outcome; this is the input — whether the pages those answers are built from can be fetched, parsed and quoted at all.
Free · no account
Run the fifteen checks on your own domain
The report states what was measured, what we saw, and what to change. If a check cannot be measured, it says so instead of guessing.