How to check llms.txt and whether GPTBot can crawl you

A lot of AEO pitches start with a score. Start with two URLs instead: yoursite.com/llms.txt and yoursite.com/robots.txt. If the first is missing, crawlers have no official map. If the second blocks GPTBot, ChatGPT never sees the map — or the rest of the site — unless it already has you from somewhere else.

We shipped live checkers for both. llms.txt fetches the file from our server. GPTBot / robots.txt parses Allow vs Disallow. The crawl checklist stacks those with ai.txt and sitemap, still without a score. JSON-LD lists schema.org types on a page, including FAQPage.

What llms.txt actually is

A plain-text (usually Markdown) file at the site root. Same job as robots.txt, aimed at models: who you are, which URLs to read first. It is not in Google’s ranking systems. It does not replace a sitemap. It does not make ChatGPT cite you.

A useful file is short and honest: product in one paragraph, then links to pricing, docs, and the pages you would want a site: query to land on. A 4,000-word marketing dump is worse than nothing.

Then check whether GPTBot is allowed

OpenAI publishes several user-agents. The ones that matter for “can ChatGPT’s crawler hit me” are GPTBot (training / indexing crawl), ChatGPT-User (user-initiated fetches), and OAI-SearchBot (Search). ClaudeBot, PerplexityBot, Google-Extended, and Googlebot are the other names teams usually care about.

An empty robots.txt, or no file at all, generally means allowed. A GPTBot group with Disallow: / means blocked. A * group can also block GPTBot if there is no more-specific GPTBot group. Longest matching rule wins. Guessing from a blog post is how people lock themselves out.

What we fetch — and what we will not say

• llms.txt checker: HTTP status, file vs a homepage pretending to be text, first kilobytes.
• AI crawler checker: seven agents vs / on the live robots.txt, plus Sitemap: lines.
• AI crawl checklist: those plus ai.txt and sitemap — pass, fail, or note.
• JSON-LD checker: types in script tags. FAQPage is a flag, not a grade.

We do not invent a 0–100 AEO score from this. Pass/fail on files you can open in a browser is the whole product. Paid visibility monitors are for scheduled prompts across engines — a different job.

After the files are sane

Then measure mentions: the questions buyers actually ask, who gets named, which URL is cited. That is AEO, not a root-file checklist. Manual loop in how to track ChatGPT mentions. Tool shortlist in the Index, not in this memo.