# llms.txt and AI search visibility for sites

What llms.txt is for, how it differs from robots.txt, and a practical file for a developer or store site.

- Date: 2026-08-22
- Category: SEO

llms.txt is a markdown file at a path on your site that tells language-model agents what this site is and where the clean documents are. It is not robots.txt, it is not a sitemap, and it will not rank a Magento SKU by itself. This note is for developer sites, CMS sites and stores that already have honest HTML, and want a curated map for agents that fetch on demand.

I am Alan Vo. I ship Gold Coast production sites, including Drupal and Laravel properties that have to be readable by people first. The [llms.txt proposal](https://llmstxt.org/) is the spec I follow: an H1, a short summary, then lists of links to markdown (or equally clean) URLs. My own site uses that pattern because the HTML is already built from markdown. A store that only has a JavaScript catalogue still needs fetchable product HTML before an llms.txt link means anything.

## What llms.txt is for in 2026

llms.txt is for giving agents a small, curated index of expert pages, so they do not have to guess from a 4MB homepage. The proposal exists because HTML is wrapped in navigation and scripts, context windows are finite, and agents already fetch documentation when they answer a question about a product or a person.

Jeremy Howard’s spec (v2) is deliberately simple. The file is markdown. The only required section is an H1 with the site or project name. A blockquote summary is recommended. Then optional prose. Then H2 sections that contain lists of links with short notes. An “Optional” section is for links an agent can skip when context is tight.

It can live at `/llms.txt` for the whole origin, or at a path like `/docs/llms.txt` for that subtree. Where more than one file applies, the most specific path should win. That matters on Magento or Drupal if you only want to describe `/blog/` or `/docs/` and not every filtered category URL.

The spec also encourages markdown twins of important pages (`page.md` or `index.md` next to the HTML) and `rel="alternate" type="text/markdown"` plus `rel="describedby"` pointing at the llms.txt that covers that path. My portfolio build already emits markdown alongside HTML for work and notes. That is the honest version of the idea. A WordPress plugin that generates llms.txt without markdown bodies is a table of contents to HTML that the agent still has to clean.

What it is not:

- Not a robots allow list. [Magento robots.txt for AI crawlers](https://alanvo.com/blog/magento-robots-txt-ai/) (and the same file on any stack) still decides whether a bot may fetch. llms.txt assumes the agent is allowed to retrieve the links you publish.
- Not sitemap.xml. Sitemaps list everything indexable for search engines. llms.txt should list a few dozen useful documents, not 40,000 SKUs.
- Not a guarantee that ChatGPT or Gemini will cite you. It is a courtesy and a machine-readable front door. Adoption is real on docs sites. It is not a ranking factor Google publishes.

![llms.txt research on a laptop with search results](../../images/blog/llms-txt-ai-search-1.jpg)

## Why llms.txt showed up on production sites

llms.txt showed up on production sites because agents fetch constantly, and a government or school CMS is a maze if you only offer HTML. Documentation teams felt it first. Personal and organisation sites are next, which is why a CV site or a regulator site benefits more than a faceted Magento grid.

In 2024 the file was a proposal. By 2026 docs platforms generate it, Lighthouse has agentic checks that look for it, and labs publish their own developer llms.txt. That is social proof for the format. It is not a reason to auto-dump every Magento category into markdown.

The production question I ask: would an agent with only this file be able to answer “who is this for, what is for sale or published, and where is the canonical page?” If the answer is no, the file is a decoration.

Stores can still use it. A short llms.txt that points at shipping policy, returns, size guide, and a handful of pillar collections is useful. A file that lists every PDP will not fit a context window and will go stale the first time merchandising runs a sale.

Public-sector and education sites gain more. Information architecture is the product. An llms.txt that points at the quality and safety hub, the admissions path, or the news index is a kindness to agents that would otherwise scrape the mega menu.

## How to write llms.txt for a developer or store site

You write llms.txt by naming the site, summarising it in one blockquote, linking only pages that already exist as clean documents, and keeping the file small enough to read in one pass. Then you make sure robots.txt allows those URLs.

A practical shape for a personal developer site follows the spec in order. The first heading in the file is the site name. A blockquote summarises who you are in one or two sentences. A short paragraph can add contact and what you ship. Then file lists: a Work list with markdown links to case studies, a Notes list with a few blog URLs, and an Optional list for secondary pages such as a colophon. Each list item is a markdown link, a colon, and a clause about why an agent should fetch it. Example destinations I would actually include: the [NDIS Commission Drupal work in markdown](https://alanvo.com/work/ndis-commission/index.md), an [on-page SEO note in markdown](https://alanvo.com/blog/on-page-seo-ecommerce/index.md), and a school IA page if that is the property you are describing.

![Notebook outline for headings and title tags](../../images/blog/llms-txt-ai-search-2.jpg)

Use real markdown URLs if you have them. If you do not, link HTML and say so, or generate markdown. Do not invent `.md` URLs that 404. An agent that follows a 404 will not give you a second chance in that session.

For a store:

- H1 is the brand
- Blockquote is what you sell and where you ship
- One section for policies (returns, shipping, warranty) because agents get those questions wrong
- One section for a few collection hubs, not the whole catalogue
- Optional: a single “how to buy” or size guide

For Magento, do not generate llms.txt from layered navigation. You will republish the crawl-budget disaster in markdown. Point at the same indexable categories you already chose in on-page SEO.

For Drupal, a recipe or module can emit the file. Still curate. A regulator site should list the living hubs, not every PDF from 2014.

Implementation notes:

- UTF-8, no need for a BOM
- Absolute HTTPS links
- Keep descriptions one clause
- Re-generate or re-check when you add a pillar page
- `Link` headers or HTML `rel="describedby"` help clients find the file
- Cache it like any other static file. Stale llms.txt that points at renamed slugs is worse than none

Pair it with [on-page SEO for ecommerce](https://alanvo.com/blog/on-page-seo-ecommerce/) when the destination pages are products. Agents cite what they fetch. Thin collections stay thin in markdown.

## Production checklist for llms.txt

1. Decide the path: origin root, or `/docs/llms.txt`, not both saying different things about the same URLs.

2. Write the H1 and blockquote in human language. No slogan salad.

3. Link 10 to 40 URLs that already return 200. Prefer markdown.

4. Verify every link. Broken links in llms.txt are a credibility hit.

5. Confirm robots.txt allows those paths and allows the retrieval bots you care about.

![Product grid on a phone used for on-page SEO checks](../../images/blog/llms-txt-ai-search-3.jpg)

6. Add `rel="describedby"` on HTML templates or a `Link` header at the CDN.

7. If you publish markdown twins, add `rel="alternate" type="text/markdown"`.

8. Do not list checkout, cart, account, or filtered Magento URLs.

9. Do not paste the entire sitemap.

10. Re-fetch llms.txt after deploy. CDNs cache `/llms.txt` aggressively because it looks like a config file.

11. Test by asking an agent a factual question with only the file as context. If it cannot find contact or the main work, the file failed.

12. Review quarterly. Bots change. Your IA changes faster.

## What breaks llms.txt

Auto-generation from a sitemap breaks it. You recreate sitemap.xml in a worse format.

Plugin defaults that claim “every page is LLM ready” break it. Most Magento and Shopify pages are not expert documents. They are templates.

Blocking AI crawlers in robots.txt while advertising llms.txt breaks it. You invited someone and locked the door.

404 markdown extensions break it. Only advertise `.md` if the server serves it.

Stale product claims break it. llms.txt is a content surface. If it says you sell a line you discontinued, agents will say it too.

Confusing llms.txt with `noindex`. The file can link a page you still want out of Google. That is usually a mistake. Curate for both humans and agents unless you have a real reason.

Security: do not put staging URLs, internal admin paths, or unpublished case metrics in the file. It is public.

## How to measure llms.txt

You measure llms.txt by whether agents that fetch it can answer accurately, and by server logs for `/llms.txt` and the markdown URLs, not by a Google ranking report. There is no Search Console report for this file.

I look at 200s, cache hit ratio, and whether the linked markdown is smaller and cleaner than the HTML. If the markdown is a dump of the same navigation chrome, you did not do the work.

I do not claim traffic from publishing llms.txt. I claim a clearer answer when someone asks an agent about the site. For a developer portfolio that is the point. For a store, policies and hubs are the point. Catalogue SEO remains titles, crawl and schema.

## Related work on this site

llms.txt pays off on sites whose information architecture is the product. I shipped Drupal for the [NDIS Commission](https://alanvo.com/work/ndis-commission/), where quality and safety content has to be findable, not decorative, and Laravel for [Brighton College](https://alanvo.com/work/brighton-college/), a deep school IA for admissions and daily life. Those properties are what I have in mind when I say curate hubs for agents. A personal llms.txt is the same idea at a smaller scale: point at the living pages, not at every filtered URL a CMS can emit.

![Team reviewing a site map on a whiteboard](../../images/blog/llms-txt-ai-search-4.jpg)

## llms.txt FAQ

### Is llms.txt required for AI search visibility?

llms.txt is not required for AI search visibility. Retrieval bots can still fetch HTML if robots.txt allows it. The file helps agents that look for a curated map, especially on docs and content-heavy CMS sites. Stores still need crawlable product pages first.

### How is llms.txt different from robots.txt?

llms.txt is a markdown index of useful documents. robots.txt is a crawl permission file. Google’s robots rules do not read llms.txt as a Disallow list. You still need both if you care about AI crawlers and about not wasting their context.

### Should a Magento store list every SKU in llms.txt?

A Magento store should not list every SKU in llms.txt. List policies and a few collection hubs. SKUs belong in sitemaps and in on-page product documents. An llms.txt that is 2MB is a failed spec.

### Can I generate llms.txt with a WordPress or Drupal plugin?

You can generate llms.txt with a plugin, and you still have to curate the links. Auto-including every post and product reintroduces sitemap noise. Treat the plugin as a publisher, not as an editor.


![Browser window with a store collection page](../../images/blog/llms-txt-ai-search-5.jpg)


HTML version: https://alanvo.com/blog/llms-txt-ai-search/