What is llms.txt?
And do you actually need one?

llms.txt is a plain-text file you put at the root of your site that gives AI assistants a curated map of your most important pages. It is to language models what robots.txt is to search crawlers — except instead of controlling access, it offers curation: a short, human-written summary of what you are and where your best content lives, in a format a model can read in one fetch. The honest short answer to “do you need one?” — it won't hurt, it's cheap to add, and it can help assistants describe you accurately. But it is not magic, and it is not the first thing you should fix.

What llms.txt actually is

It's a proposed convention: a Markdown file served at https://yourdomain.com/llms.txt. The idea is that AI assistants have a limited budget for how much of your site they can read and reason over, so rather than making them guess from your navigation and sitemap, you hand them a clean, prioritized index.

The format is just structured Markdown:

• An H1 with your site or product name.
• A blockquote summary — one or two sentences on what you do.
• ## sections grouping links, each link followed by a short description of what that page is.
• An optional ## Optionalsection for links a model can skip if it's short on space.

Here's a minimal example — this is roughly AEOScan's own:

# AEOScan

> A free AI visibility audit. Paste a URL and it checks
> crawler access, llms.txt, JSON-LD and content structure,
> asks four assistants what they know about the site, and
> hands you copy-paste fixes.

## Key pages

- [Home](https://aeoscan.net/): Run a free AI visibility audit
- [Blog](https://aeoscan.net/blog): Guides on AEO and AI visibility

## Optional

- [Contact](mailto:hello@aeoscan.net): Questions and feedback

How it differs from robots.txt

They look similar and sit in the same place, which causes a lot of confusion. They do opposite jobs:

• robots.txt is access control — it tells a crawler whetherit may fetch a path. It's a gate.
• llms.txt assumes the model is already allowed in and tells it what's worth reading. It's a map.

This is why the order matters: an llms.txt is pointless if your robots.txt — or your CDN — is blocking the crawlers in the first place. The map does nothing if the door is locked. We covered checking crawler access here.

llms.txt vs llms-full.txt

You'll see both. llms.txt is the short index above — a summary plus links. llms-full.txt is an optional, bigger file that inlines the actual content of those pages as Markdown, so a model can ingest everything in a single request without crawling each URL. For a docs-heavy product the full file can be worthwhile; for most marketing sites, the short index is enough. Start small.

So — do you actually need one?

Here's the part most posts won't tell you straight. Adoption of llms.txt is still early, and not every assistant has confirmed it reads the file. So treat it as a cheap, forward-looking signal, not a guaranteed ranking lever. It earns its place because it costs ten minutes, it can't hurt, and when it is read it helps an assistant describe you in your own words instead of guessing.

But it sits near the end of the priority list, not the top. If you only do a few things for AI visibility, do them in this order:

1. Let the crawlers in. Allow GPTBot, ClaudeBot, PerplexityBot and friends in robots.txt andat your CDN/WAF. Nothing else matters if you're blocked.
2. Render content in raw HTML.AI crawlers generally don't run JavaScript. If your content only appears after JS, the bot sees an empty shell.
3. Ship clean structured data. JSON-LD that says what you are, in the HTML the bot actually receives.
4. Then add llms.txt.Now that you're readable, hand the model a map.

Adding llms.txt while skipping steps 1–3 is like printing a beautiful table of contents for a book nobody can open.

How to create one

You can write it by hand in a few minutes: open a text editor, follow the format above, list your five to fifteen most important pages with a one-line description each, and save it as llms.txtat your site root so it's served at /llms.txt with a 200. Keep it current when your key pages change — a stale map is worse than none.

If you'd rather not start from a blank file, AEOScanchecks whether you already have an llms.txt, flags whether it's well-formed, and generates a ready-to-paste one from your site — alongside the more important checks (crawler access, JS rendering, structured data) so you fix things in the right order. Free, about 30 seconds, no signup.

Common mistakes

• Publishing llms.txt while blocking the crawlers. The single most common one. The file is never fetched.
• Linking to JavaScript-only pages.If the page the link points to renders client-side, the model still can't read it.
• Letting it go stale. Pointing models at pages that moved or no longer exist erodes trust in the file.
• Wrong location or content type. It must be at the root and served as plain text, not buried in a subfolder or returned as HTML.

The bottom line

llms.txt is a genuinely useful, low-effort addition — a curated map that helps assistants find and describe your best content. Just keep it in perspective: it's the finishing touch after you're crawlable, readable without JavaScript, and structured. Get the door open first; then hand visitors the map.

Frequently asked questions