What is llms.txt?
llms.txt is a proposed convention, introduced in 2024 by Jeremy Howard, co-founder of fast.ai. The idea: web pages are full of menus, scripts and adverts, so give AI systems a clean, curated reading list instead.
The file uses simple Markdown: a title, a one-line summary, and sections of links with short descriptions. A related file, llms-full.txt, puts a site’s full content into one Markdown file.
| File | Purpose | Who reads it |
|---|---|---|
| robots.txt | Says which bots may crawl which pages | All major search engines and AI crawlers |
| sitemap.xml | Lists every URL you want indexed | Search engines |
| llms.txt | A curated summary and reading list for AI tools | Some AI agents and coding tools, voluntarily |
Does Google use llms.txt?
No. Google’s guide to its generative AI features, updated in July 2026, says you do not need llms.txt or any other special AI file to appear in Google Search, AI Overviews or AI Mode, because Google Search does not use them. It adds that creating one for other services is fine and will neither harm nor help your Google visibility.
One point causes confusion: Chrome’s Lighthouse tool added an llms.txt check in 2026, under its “agentic browsing” audits. Lighthouse calls llms.txt an emerging convention that can save AI agents time understanding a site, and treats it as optional. That is about AI agents browsing your site, not about Google rankings.
Do ChatGPT, Claude or Perplexity use it?
None of the major AI assistants has confirmed that its search feature uses llms.txt to choose sources. Where the file is clearly used is developer documentation: AI coding assistants fetch it to find the right pages quickly. OpenAI, for example, publishes an llms.txt for its own developer documentation.
So the honest picture is: useful for documentation-heavy and software sites, a minor nice-to-have for most others.
Should your business create one?
| Your site | Recommendation |
|---|---|
| Software, API or developer documentation | Yes. AI coding tools use it. |
| Large content or knowledge site | Worth doing, low effort. |
| Agency, consultancy or service business | Optional. Do it after the basics below. |
| Local shop or clinic | Low priority. Focus on Google Business Profile and reviews. |
The basics that matter far more for AI visibility: let AI search crawlers access your site in robots.txt, write clear service pages, keep business details consistent everywhere, and earn genuine reviews and mentions.
Free llms.txt template
Save this as a plain text file named llms.txt in your website’s root folder, and replace the example details:
# Example Agency
> Example Agency is a digital marketing agency that helps B2B companies in India grow through SEO, AI search optimisation and website development.
## Services
- [AI search optimisation](https://www.example.com/ai-seo/): Get cited in ChatGPT, Gemini, Perplexity and Google AI Overviews
- [SEO services](https://www.example.com/seo-services/): Technical SEO, content and links for organic growth
- [Website development](https://www.example.com/website-development/): Fast, search-ready business websites
## Guides
- [AEO vs GEO vs SEO](https://www.example.com/aeo-vs-geo-vs-seo/): How the three approaches differ
- [Website development cost in India](https://www.example.com/website-development-cost-india/): 2026 price ranges
## Company
- [About us](https://www.example.com/about/): Team, experience and approach
- [Contact](https://www.example.com/contact/): Request a free auditKeep it short. List your 10 to 20 most useful pages, not every URL. Update it when you add a major service or guide. Some WordPress SEO plugins can now generate the file for you.
How to publish llms.txt on a WordPress site
- Write the file in a plain text editor using the template above, and save it as llms.txt.
- Upload it to your site’s root folder, the same folder that contains wp-config.php, using your hosting file manager or FTP. Alternatively, use an SEO plugin that offers an llms.txt option.
- Check it loads at yoursite.com/llms.txt in a browser.
- Confirm it returns plain text without errors. If you use a CDN, purge its cache after uploading.
- Set a reminder to review it whenever you add a major service, guide or product.
What to include and what to leave out
| Include | Leave out |
|---|---|
| A one-sentence summary of who you are and what you do | Marketing slogans and superlatives |
| Your main service or product pages | Every blog post and tag page |
| Your most useful guides and pricing pages | Thin, outdated or duplicate pages |
| About, contact and policy pages | Private, login-only or staging URLs |
| Short, factual descriptions for each link | Keyword lists |
A better use of the next hour: the AI-readiness checklist
If you have limited time, these steps matter far more for AI visibility than llms.txt:
- Check robots.txt and your firewall allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot.
- Confirm your key pages show their main text in the raw HTML, not only after JavaScript runs.
- Rewrite your About page and top service pages with a clear one-sentence description.
- Make your name, address, phone and services identical on your site, Google Business Profile and directories.
- Add Organization schema with links to your official profiles.
- Verify your site in Google Search Console and Bing Webmaster Tools.
- Ask your three most recent happy clients for detailed reviews.
Then add llms.txt as a quick extra.
Why the debate continues
llms.txt keeps coming up because it’s cheap to create and promises a simple fix for a complex problem. Google’s own teams have sent mixed signals: Google Search says it ignores the file, while Chrome’s Lighthouse treats it as an emerging convention for AI agents. Both can be true. The file may become more useful as AI agents browse the web on users’ behalf. For search visibility today, it isn’t a lever.
What to list, by type of website
| Website type | Sections worth including |
|---|---|
| Software or API product | Getting started, API reference, authentication, key guides, changelog, pricing |
| Agency or consultancy | Services, case studies, pricing or engagement models, key guides, about and contact |
| Ecommerce store | Main categories, shipping and returns policies, size or buying guides, contact |
| Local business | Services, prices, area served, opening hours page, booking and contact |
| Publisher or education site | Topic hubs, most-used guides, editorial policy, author pages |
Describe each link in one factual line, so an AI tool knows what it will find before it fetches the page.
llms-full.txt: should you bother?
llms-full.txt places your site’s full text in a single Markdown file so an AI tool can read everything at once. It’s useful for documentation sites where developers ask AI coding assistants detailed questions. For most business websites it adds maintenance work, because every content change must be copied into the file, with little benefit. Skip it unless you run substantial documentation.
How to tell whether anything reads your llms.txt
Check your server access logs, or your CDN’s analytics, for requests to /llms.txt. Note the user agents making those requests. If, after a few months, almost nothing requests the file, that tells you how much effort it deserves for your site. Compare it with requests from AI search crawlers to your actual pages, which matter more.
Common llms.txt mistakes
- Treating it as an SEO tactic, expecting rankings or AI Overview appearances to change.
- Listing hundreds of URLs, which defeats the purpose of a curated reading list.
- Letting it go stale, with links to pages that redirect or no longer exist.
- Including private or staging URLs that shouldn’t be public.
- Using it instead of robots.txt to try to control or block AI crawlers.
- Spending agency budget on it while basic crawler access, rendering and content problems remain.
A short timeline of llms.txt
| When | What happened |
|---|---|
| 2024 | Jeremy Howard proposed llms.txt as a Markdown reading list for language models, published at llmstxt.org |
| 2025 | SEO plugins and many documentation sites began offering or publishing the file |
| May 2026 | Chrome’s Lighthouse added an optional llms.txt check under its agentic browsing audits |
| May to July 2026 | Google published and updated its guide to generative AI search, stating that Google Search ignores llms.txt |
Our recommendation
Create a short llms.txt if you have 15 minutes and a clear list of your most useful pages. Don’t pay for it as a standalone service, don’t expect it to change your Google rankings, and don’t let it distract from the work that does move AI visibility: crawler access, clear answer-first pages, consistent business details, structured data and genuine third-party mentions. Revisit the file once or twice a year, or whenever AI companies publish clear guidance on using it.
Where llms.txt is genuinely useful: documentation sites
llms.txt makes the most sense for software products with large documentation, API references or help centres. Developers often paste documentation into AI coding assistants or point an assistant at a docs URL, and a clean index of markdown pages is easier to work with than a site full of navigation, scripts and cookie banners. That’s why the files appear mostly on developer documentation, including Anthropic’s own docs, and why several documentation platforms can now generate llms.txt automatically.
If you run a SaaS product or an API, a useful llms.txt:
- Lists the getting-started guide, authentication, core concepts and the most-used API endpoints first.
- Links to plain markdown versions of each page, where your docs platform provides them.
- Groups pages under clear headings such as Guides, API reference, SDKs and Changelog.
- Points to an llms-full.txt file if your documentation is small enough to fit in one file.
- Is regenerated automatically whenever the docs change, so it never lists retired pages.
For a local business, an agency or an online store, the case is much weaker. Your customers are not loading your website into a coding assistant, and the major AI search engines have not said they use the file. Spend the time on crawlable pages, clear answers and consistent business facts first, then add a short llms.txt if you want one.
Frequently asked questions
Is llms.txt the same as robots.txt?
No. robots.txt controls which bots may crawl your pages. llms.txt only suggests what to read. To allow or block AI crawlers, use robots.txt.
Can llms.txt block AI from using my content?
No. It has no blocking function. Use robots.txt rules for bots such as GPTBot or ClaudeBot, plus firewall rules if needed.
Will llms.txt improve my Google rankings?
No. Google says Search ignores it, so it has no effect, good or bad, on rankings or AI Overviews.
Does llms.txt help me appear in ChatGPT?
There is no confirmed evidence that ChatGPT search uses it to pick sources. Allowing OAI-SearchBot in robots.txt and publishing clear, specific pages matters far more.
What is llms-full.txt?
An optional companion file that contains your site’s full text in Markdown, so an AI tool can read everything in one request. It is mostly used for documentation sites.
Where should the llms.txt file be placed?
At the root of your domain, so it loads at yoursite.com/llms.txt, in the same location as robots.txt.
Is llms.txt an official web standard?
No. It’s a proposed convention published at llmstxt.org. Chrome’s Lighthouse describes it as emerging, and adoption by AI companies is voluntary and limited.
Can llms.txt hurt my SEO?
Google says it neither helps nor harms your rankings in Google Search. Just make sure it doesn’t list private or outdated pages.
Do AI agents use llms.txt?
Some AI agents and coding tools can fetch it to understand a site quickly, which is why Chrome’s Lighthouse includes an optional check. Use by mainstream AI search assistants hasn’t been confirmed.
How long should an llms.txt file be?
Short: a title, a one-line summary and roughly 10 to 30 well-described links for most business websites. Documentation sites may list more.
Will llms.txt become more important in the future?
Possibly. As AI agents increasingly browse websites on people’s behalf, a clean summary of a site could become more useful. For now, treat it as a low-cost experiment and watch for official guidance from AI companies.
Does llms.txt help with Perplexity or Claude?
Neither company has confirmed that its search feature uses llms.txt to choose sources. Allowing PerplexityBot and Claude-SearchBot in robots.txt, and publishing clear, specific pages, are the confirmed routes into their results.
Want a full AI-readiness check, not just one file? Get a free AI visibility audit.
Sources
- Google Search Central: Optimizing your website for generative AI features (updated 10 July 2026)
- Chrome for Developers: llms.txt audit in Lighthouse (updated 5 May 2026)
- llmstxt.org: The llms.txt specification
- OpenAI developer documentation (links to its own llms.txt)
