SEOMaster
Can AI assistants read your website
Find out whether ChatGPT, Claude, Perplexity and AI Overviews can read your site — and whether you are allowing them to. We analyse AI crawler rules in robots.txt, content available without JavaScript, HTML structure, structured data, the llms.txt file, and whether an author and date are present.
What the AI readiness test covers
Your policy towards AI crawlers — robots.txt rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended and others. We separate training crawlers from search crawlers, which fetch a page in order to cite it in an answer with a link. No rules means full access, and we score that well — points come off only for blocking the search crawlers.
Whether the content can be extracted at all — A semantic container for the main content (<main> or <article>) and the ratio of text to markup — plus, when the renderer has run, a comparison of the raw HTML against the post-JavaScript version. Most AI crawlers do not execute JavaScript and receive an empty shell.
Structured data — Presence and completeness of JSON-LD: a schema describing the entity (Organization, LocalBusiness or Person) and one matching the content (Article, Product, FAQPage, HowTo). These let a machine read a fact directly instead of inferring it from prose.
llms.txt, author and date — An /llms.txt file pointing models at your most important pages — a 2024 proposal no major provider honours yet, so its absence barely moves the score. Plus an author and a publication or update date: an assistant deciding whether to cite a source looks for the same things a person verifying a claim would.
We measure whether an assistant can read the page and whether it is permitted to. We do not measure, and will not promise, placement in AI answers — that cannot be established from the outside, because models publish neither their sources nor their selection criteria. If a tool hands you a "ChatGPT visibility score", it is worth asking where the number came from. Weightings and thresholds are in the methodology.
Training versus search — the distinction that decides your traffic
AI crawlers fall into two groups with completely different consequences for you, and most advice lumps them together and tells you to block everything.
Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) fetch content to use in model training. They send nobody your way. Blocking them is a perfectly reasonable business decision and costs no traffic.
Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) fetch a page at the moment someone asks a question, and cite it in the answer with a link. These are the ones that bring visitors. Blocking them is the exact equivalent of opting out of search results.
That is why this test only deducts points for blocking search crawlers. Blocking training crawlers is neutral — it is your choice, not a fault.
How to set this deliberately
1. Block training, keep citation Usually the sensible compromise: your content does not feed model training, but can still be cited with a link when somebody asks a question about your subject.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /2. Wrap your content semantically Main content in <main> or <article>, navigation in <nav>, footer in <footer>. Without these a crawler is guessing where the menu ends and the article begins. The same change improves accessibility, so it counts twice.
<main>
<article>
<h1>Title</h1>
<p>The content you want cited…</p>
</article>
</main>3. Render content on the server After crawler policy this is the heaviest single signal in the pillar. Google will usually cope with JavaScript; AI assistants usually will not. If your content only appears after scripts run, the page is empty as far as they are concerned. SSR or static generation removes the problem at source.
4. State the facts directly in JSON-LD Organization or LocalBusiness for the company, plus a type matching the page's content. A machine then reads your name, address or author as data rather than inferring them from a paragraph.
{"@context":"https://schema.org","@type":"Organization","name":"Your Company","url":"https://yourdomain.com"}5. Add an author and a date A model choosing which source to cite looks for credibility signals in exactly the places a human would. An author and a last-updated date are the cheapest of them.
<meta name="author" content="Jane Doe">
<meta property="article:published_time" content="2026-09-22">What this test will not tell you
It will not tell you whether ChatGPT cites your site, or in what position. Models publish no source registry, and the answer to the same question varies between sessions and regions. Any "AI visibility score" is therefore either a sample of a few prompts or simply invented.
It also does not judge the quality or truth of your content. We measure the technical layer: access, extractability, structure and attribution. Whether the content deserves to be cited remains an editorial question.
Finally, llms.txt is a proposal, not a standard. We include it because it costs nothing and may prove useful, but deliberately at a low weight — no major provider honours it today.
Frequently asked questions
Should I block AI crawlers? It depends which. Blocking training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot) is a defensible decision that costs no traffic, because they send nobody to you. Blocking search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) cuts off cited links, which are real visits. The distinction matters more than the decision itself.
Does structured data improve AI visibility? It helps a machine read a fact directly rather than inferring it from prose, which reduces the risk of your name, address or authorship being garbled. It is not a visibility switch, and nobody has demonstrated that models choose sources based on the presence of JSON-LD. Treat it as hygiene, not leverage.
My site runs on JavaScript. Is that a problem? For AI assistants, usually yes. Google has rendered JavaScript for years and will normally manage, but most AI crawlers fetch raw HTML and execute no scripts — they see an empty shell. Server-side rendering or static generation removes the problem at source.
Do I need llms.txt? It is not required. It is a 2024 proposal that no major AI provider honours yet, which is why it carries a low weight here and its absence barely affects the score. It costs a few minutes and may pay off if the standard catches on, but it is not the first thing to do.
Is the test free, and do I need an account? The test is free and requires no login or email address. AI readiness is one of the eight pillars in a full SEOMaster audit.