llms.txt & Markdown Files: Do They Actually Help Your AI Visibility?

Every few months, the SEO world gets a new "game-changer" that turns out to be mostly noise. When llms.txt started getting traction in late 2024, I was sceptical. After testing it across 8 client sites over 3 months, I have a more nuanced view. Here's what I found — and what I think you should actually do.

What is llms.txt?

llms.txt is a proposed standard — not an official protocol — for helping large language models understand your website more effectively. The idea is simple: you place a plain-text or markdown file at yourdomain.com/llms.txt that provides a curated summary of what your site contains, which pages are most important, and how you'd like AI engines to understand your brand.

It was proposed by Jeremy Howard (of fast.ai fame) in late 2024 and gained rapid traction in the developer and SEO community. The protocol includes two files:

• /llms.txt — a brief summary of your site and its key pages, in markdown format
• /llms-full.txt — a comprehensive version with full content of key pages, for AI engines that want more context

How does it compare to existing standards?

llms.txt vs robots.txt

robots.txt tells crawlers what not to index. llms.txt tells AI engines what to prioritise. They're complementary, not competing. robots.txt is a hard technical instruction; llms.txt is a soft recommendation that AI engines can choose to honour or ignore.

llms.txt vs sitemap.xml

A sitemap tells search engines all your pages exist. llms.txt tells AI engines which pages matter most and what they contain. The sitemap is breadth; llms.txt is curation.

What does an llms.txt actually look like?

Here's a simplified example of what an effective llms.txt for an AI SEO consultant might look like:

# Sagar Dubey — AI SEO Consultant

> I help SaaS, D2C, and B2B brands rank in Google and AI search engines 
> through data-driven SEO, programmatic content, and LLM optimisation.

## Core Services
- [AI SEO Audit](/services/ai-seo-audit): Comprehensive 40-point audit
- [Programmatic SEO](/services/programmatic-seo): Scale to 1000+ pages
- [AEO / LLM Visibility](/services/aeo): Get cited in Perplexity, Claude, ChatGPT

## Key Resources  
- [Blog](/blog): Practical guides on GEO, AEO, and AI search
- [Case Studies](/results): Real client results with measurable outcomes
- [Contact](/contact): Book a free 30-minute audit call

My test results across 8 client sites

I implemented llms.txt on 8 client sites between October 2024 and January 2025, with 4 control sites that did not implement it. My findings:

• Perplexity — Marginal improvement in citation accuracy (the pages cited were more often the pages I'd listed as priority in llms.txt). Not a dramatic citation rate improvement, but better targeting.
• ChatGPT (browsing mode) — No measurable difference. ChatGPT appears to crawl pages directly rather than consulting the llms.txt file.
• Claude — Inconclusive. Claude's responses are largely from training data, not real-time retrieval, so llms.txt impact is minimal for base model responses.
• Google AI Overviews — No observable difference. Google's system relies on its own crawl and ranking signals, not llms.txt.

Getting started with llms.txt

1. Create a plain text file called llms.txt in your site root
2. Write a 2–3 sentence description of what your site is about and who it serves
3. List your 5–10 most important pages with URLs and brief descriptions
4. Use standard markdown formatting (# for headers, - for bullets, [text](url) for links)
5. Optionally create llms-full.txt with more comprehensive content
6. Submit the URL to Perplexity's index tool if/when they make it available

The markdown files question

Separately from llms.txt, some technical SEOs are publishing markdown versions of their key pages (e.g., /about.md, /services.md) as AI-friendly alternatives to HTML. The theory is that markdown is closer to the training data format that AI models learn from.

My view: this is worth testing for high-priority pages, especially documentation and guides. GitHub's prevalence in AI training data means markdown formatting does have a familiarity advantage. But it's a minor signal, not a fundamental one.

Frequently Asked Questions