llms.txt: How to make your website readable for AI

Dark blue graphic with an orange target circle and the title “llms.txt”

An llms.txt is a small Markdown file in the root directory of your website that explains to AI systems in compact form what the site is about and which content matters. It makes your website easier to read for language models and AI agents, but according to Google itself it does not improve your rankings in Google Search. In this post, you will learn how the file is structured, who uses it and how to create it in a few steps.

Key takeaways

  • The llms.txt is an open proposal by Jeremy Howard (Answer.AI) from September 2024, not an official web standard.
  • According to the specification on llmstxt.org, only an H1 heading with the name of the website is required; everything else is optional.
  • Google Search does not use the file, but it does no harm either.
  • Chrome Lighthouse now checks the llms.txt in the “Agentic Browsing” category.
  • In WordPress, plugins such as Yoast SEO can create the file in a few minutes.

What is an llms.txt?

An llms.txt is a Markdown text file that gives language models a short overview of your website with links to the most important content. It was proposed by Jeremy Howard of Answer.AI on 3 September 2024. His basic idea: compiling the right context for a language model from a normal website with navigation, ads, scripts and cookie notices is ambiguous. A curated list of the relevant content in a consistent format takes this work off AI systems’ hands.

Important for context: the llms.txt is a proposal from the community, not a standard from a standards body. The specification is published and developed further on llmstxt.org. How much individual AI providers use the file is up to them.

How is an llms.txt structured?

According to the specification, the file follows a fixed order: title, short description, optional explanations and link lists under H2 headings. Only the H1 heading is required. The elements in their order:

  1. H1 with the name of the website or project (the only required part).
  2. Blockquote with a short summary of what the website offers.
  3. Free-form explanations in paragraphs or lists, but without further headings.
  4. Link lists under H2 headings, each entry as a Markdown link, optionally with a colon and a short note.
  5. “Optional” section for secondary links that an AI system may skip when context is limited.

A simplified example for an agency website could look like this:

# Sample Agency
> Agency for SEO and websites in Sample City.

We advise small and medium-sized businesses.

## Services
- [SEO](https://www.example.com/seo/): Search engine optimisation
- [Websites](https://www.example.com/websites/): Planning and implementation

## Optional
- [Blog](https://www.example.com/blog/): Articles

The specification also recommends providing important pages as a plain Markdown version, for example at the same address with the extension “.md”. This is common for documentation but not a must for typical business websites.

How does llms.txt differ from robots.txt and sitemap.xml?

All three files usually sit in the root directory but serve completely different purposes: control, list and explain. The robots.txt defines, under the RFC 9309 standard, which areas crawlers may access. The sitemap.xml lists all addresses that should be indexed. The llms.txt, on the other hand, describes content and sets priorities.

FilePurposeFormatStatus
robots.txtAllow or block crawler accessText with rulesStandard (RFC 9309)
sitemap.xmlList all indexable addressesXMLSupported by search engines
llms.txtSummarise and prioritise content for AIMarkdownProposal, voluntary

So an llms.txt does not replace either of the other two files. If you block AI crawlers, an llms.txt achieves little, because the actual pages remain inaccessible.

Does Google use llms.txt for rankings or AI Overviews?

No, according to its own documentation, Google Search does not use the llms.txt, neither for classic rankings nor for AI Overviews. In its guide to generative AI features in Google Search, Google writes that you do not need new machine-readable files, AI text files or Markdown versions to appear in Google Search. At the same time, Google makes clear that it is perfectly fine to create an llms.txt for other services. It neither helps nor harms visibility in Google Search.

For Google Search, the familiar basics still count: crawlable and indexable pages, helpful content with a clear structure and structured data as part of your SEO strategy. Read how AI answers work on Google in our post on Google AI Overviews.

What is an llms.txt still worth it for?

An llms.txt is useful above all for AI agents and assistants outside Google Search that retrieve websites specifically and need to understand them quickly. Chrome Lighthouse now checks the file in the “Agentic Browsing” category. According to the Chrome documentation, agents without an llms.txt may have to crawl more of your website to understand its structure and main content.

  • AI agents: assistants that compare providers or gather information on behalf of users find the relevant pages faster via the file.
  • Developer tools: coding assistants often read the llms.txt of software documentation.
  • Your own control: you decide which services and content count as particularly important.
  • Low effort: a simple version can be created quickly and is easy to maintain.

Whether and how often individual AI providers such as ChatGPT, Claude or Perplexity actually retrieve the file is not reliably documented in public. So see the llms.txt as an inexpensive addition to your AI visibility strategy, not as a shortcut.

How do you create an llms.txt in 5 steps?

For a solid llms.txt, all you need is a text editor, a list of your most important pages and access to the root directory of your website. Here is how to proceed:

  1. Select core pages: home page, service pages, about us, contact and your best guides. Less is more here.
  2. Write a short description: one or two sentences on who you are, what you offer and for whom.
  3. Write the file: H1, blockquote, link lists under H2 headings and an “Optional” section, saved as UTF-8.
  4. Upload: place the file in the root directory so that it is available at your domain with “/llms.txt”.
  5. Check and maintain: test retrieval in the browser, run Lighthouse and update the file when you add services or addresses change.

How do you set up llms.txt in WordPress?

In WordPress, SEO plugins generate the llms.txt automatically, so you do not have to upload a file via FTP. In Yoast SEO, you activate the feature in the settings under site features in the AI tools area, as described in the Yoast help. There you can also choose manually which pages appear in the file. Yoast notes that the feature is not available on multisite installations.

Our free plugin GEO Score for WordPress checks whether your website as a whole is well prepared for AI systems. Besides the llms.txt, it also evaluates structured data, answer formats and access for AI crawlers.

Practical example: a trades business with a WordPress website activates the llms.txt in its SEO plugin. The automatically generated version initially lists all pages, including the legal notice and old promotion pages. After manual selection, the file only contains the home page, services, service area, contact and the three most-read guides. An AI agent can then see at a glance what the business offers and where.

Which llms.txt mistakes should you avoid?

Common mistakes are overloaded link lists, outdated addresses and content that does not match the actual website. Pay particular attention to these points:

  • Linking everything: a copy of the sitemap misses the point. Prioritise deliberately.
  • Dead links: after a relaunch, old entries often point to redirects or error pages.
  • Marketing language: AI systems need factual descriptions, not slogans.
  • Server errors: if retrieval returns an error, Lighthouse marks the page as failing, according to the Chrome documentation.
  • Blocked crawlers: if the robots.txt blocks AI crawlers, even the best llms.txt is of little use.

We explain how to structure your content so that search engines and AI understand it in our SEO Basics.

Frequently asked questions

Is an llms.txt mandatory?

No, an llms.txt is voluntary. It is an open proposal from 2024, not a binding web standard. No search engine and no law requires the file. Chrome Lighthouse therefore does not rate a missing llms.txt as an error but as “not applicable”. It is still useful if you want to make it easier for AI agents to access your most important content and do not mind the small maintenance effort.

Does an llms.txt improve my Google ranking?

No. In its documentation on generative AI features, Google states explicitly that Google Search does not use such files. An llms.txt neither helps nor harms your visibility in Google Search, including in AI Overviews. For better rankings, crawlable pages, helpful and well-structured content, clean technology and structured data still count. The llms.txt is aimed at other AI systems and agents.

Where does the llms.txt need to be located?

The usual place is the root directory of your domain, i.e. directly at your address with “/llms.txt” at the end. For a typical business website, a single file in the root directory is enough. It is important that it can be accessed without logging in and that the server does not return an error.

How often should I update the llms.txt?

Update the file whenever your offer or important addresses change, for example with new services, a relaunch or new guides. If a WordPress plugin generates the file automatically, briefly check after such changes whether the selection still fits. Regardless of this, we recommend a quick look during every major content planning session so that no dead links or outdated descriptions remain in the file.

Do I also need Markdown versions of my pages?

For most business websites, no. The specification recommends Markdown versions with the extension “.md” mainly for content that AI agents should read in detail, such as technical documentation. On a normal website with clean HTML, clear headings and little clutter, AI systems manage without an additional version. Focus first on well-structured content and a well-maintained llms.txt.

Make your website ready for AI search

We check how well AI systems understand your website, set up llms.txt, structured data and crawler access, and build your content in an answer format. We have supported businesses in digital marketing since 2003.

Dipl.-Ing. Rida Mekaoui

About the author

Dipl.-Ing. Rida Mekaoui has worked in digital marketing since 2003 and leads Digital Targets in Larnaka, Cyprus, as founder and managing director. He advises businesses across Europe on SEO, AI visibility, Google Ads and e-commerce.

More about us →