Free llms.txt Generator
Generate a spec-compliant Markdown index for AI agents and LLMs. Crawls JavaScript-rendered sites, detects frameworks, and structures clean content summaries.
An llms.txt file is a lightweight Markdown document placed at your site's root (/llms.txt) that provides AI agents and large language models with a clean, curated index of your authoritative content. Unlike traditional scrapers, our generator renders JavaScript sites, detects web frameworks, and structures clean Markdown summaries tailored for AI discovery.
What a Valid llms.txt File Looks Like
The official standard requires clean Markdown: a single H1 header, a blockquote summary, optional context, and categorized link lists formatted as - [Title](URL): description.
# SEOWebGrow > Free technical SEO tools, Generative Engine Optimization utilities, and developer guides. SEOWebGrow provides free browser and API tools to help engineers, developers, and search marketers optimize content for both traditional search engines and autonomous AI agents. ## Core Tools - [Free llms.txt Generator](https://seowebgrow.com/tools/llms-txt-generator): Crawl JavaScript sites, detect CMS frameworks, and generate spec-compliant Markdown files. - [llms.txt Validator](https://seowebgrow.com/tools/llms-txt-validator): Test existing llms.txt files against Lighthouse agentic browsing audit criteria. - [Free XML Sitemap Generator](https://seowebgrow.com/tools/sitemap-generator): Generate search engine sitemaps with clean URL discovery and priority tags. - [Keyword Difficulty Analyzer](https://seowebgrow.com/tools/keyword-analyzer): Analyze search competition and SERP difficulty metrics. ## Documentation & Guides - [Generative Engine Optimization Guide](https://seowebgrow.com/generative-engine-optimization): Strategic playbook for ranking across ChatGPT, Claude, and AI Overviews. - [The llms.txt Complete Guide](https://seowebgrow.com/blog/llms-txt-complete-guide): In-depth tutorial on formatting, deployment, and agentic workflows. - [llms.txt vs robots.txt Comparison](https://seowebgrow.com/blog/llms-txt-vs-robots-txt): Architectural breakdown of crawler access control versus AI context delivery. ## Optional - [The llms.txt Google Contradiction](https://seowebgrow.com/blog/llms-txt-google-contradiction): Server-log evidence and the difference between search indexing and agentic browsing. - [Agentic SEO Architecture](https://seowebgrow.com/blog/agentic-seo-llms-txt-agents-txt): Comparing llms.txt with agents.txt and WebMCP protocols.
How to Generate and Deploy Your llms.txt File
Watch how our multi-strategy engine parses sitemaps, renders JavaScript Single Page Applications, and outputs spec-compliant Markdown files.
What Does the llms.txt Specification Require?
The llmstxt.org specification defines a standardized, human-readable Markdown format designed to fit comfortably within large language model context windows. The format specifies six structured elements in a precise sequence:
1H1 Heading (Strictly Required)
The only mandatory element in the entire specification. It must be an # Title declaring the project or brand name.
2Blockquote Summary
Immediately follows the H1 (> Summary). Provides an executive summary of the site's utility for LLMs.
3Information Paragraph
An optional short paragraph providing additional architectural context, developer notes, or usage permissions.
4Section H2 Headings
Markdown headers (## Section Name) that organize related links into logical clusters like Tools, Docs, or Guides.
5Link Lists with Descriptions
Items formatted as - [Title](URL): description. Descriptions should be concise and explain what the page covers.
6Optional Section (## Optional)
A designated H2 header at the bottom for comprehensive references, changelogs, or large files that models may safely skip if context windows are full.
Crucial Spec Rule: The llms.txt standard does not support crawler blocking directives like User-agent, Allow, or Disallow. Crawl permissions belong exclusively in your robots.txt file.
What Lighthouse Checks in an llms.txt File
Starting in Lighthouse 13.3 (released May 2026), Google Chrome introduced the Agentic Browsing audit suite. If your server returns an /llms.txt file, Lighthouse automatically inspects it against three strict criteria:
1. File Begins with an H1 Heading
The audit verifies that the document opens with a level-1 Markdown heading (# Title). Files that start with plain paragraphs, bot instructions, or comments immediately fail.
2. Contains Valid Markdown Links
All linked pages must use standard Markdown hyperlinking syntax ([Anchor](URL)). Bare URLs pasted without link markdown fail the audit.
3. Minimum Length Threshold (>= 50 Characters)
The file must contain at least 50 characters of meaningful content. Stub files, placeholder comments, or 0-byte responses trigger an immediate audit failure.
/llms.txt file, Lighthouse simply skips the check. Having no file is not marked as an error or SEO penalty. Documented in detail by DebugBear's Agentic Browsing documentation .llms.txt v2 (August 2026): What Changed in the Specification?
Published on llmstxt.org/changes.html , the August 2026 v2 revision solved key discovery limitations for websites with deep directories or dynamic rendering. The update introduced four foundational enhancements:
1. Markdown Alternate Discovery
Introduced rel="alternate" type="text/markdown", allowing any HTML page to point directly to its clean Markdown twin without content-negotiation hacks.
2. Covering File Declaration
Pages declare which llms.txt covers them using rel="describedby" in an HTML <link> tag or an HTTP Link header.
3. Scoped Subpath Files
Subpaths can provide their own localized file (e.g. /docs/llms.txt). When resolving directives, the most specific subpath file wins.
4. Dual File Extensions
Both page.md and page.html.md naming conventions are officially accepted for alternate Markdown endpoints.
Copy-Paste v2 Implementation Snippets
<!-- Add to the <head> of your HTML pages --> <link rel="describedby" href="/llms.txt"> <link rel="alternate" type="text/markdown" href="/docs/guide.md">
Link: </llms.txt>; rel="describedby", </docs/guide.md>; rel="alternate"; type="text/markdown"
// next.config.mjs
/** @type {import('next').NextConfig} */
const nextConfig = {
async headers() {
return [
{
source: '/:path*',
headers: [
{
key: 'Link',
value: '</llms.txt>; rel="describedby"',
},
],
},
];
},
};
export default nextConfig;Does llms.txt Work? What the Data and Search Engines Say
When evaluating whether to implement an llms.txt file, it is vital to examine empirical research rather than marketing hype. Here is what primary industry studies and search engine guidelines reveal:
137,210 Domains Analyzed
Ahrefs examined 137K domains and found that while adoption climbed to 28% among developer documentation sites, 97% of published llms.txt files received zero crawler requests from major AI bots during their monitoring window.
300,000 Domains & AI Citations
SE Ranking analyzed 300,000 websites (reported by Search Engine Journal ) to determine if having an llms.txt file correlates with citation frequency in ChatGPT, Claude, or Perplexity. They found no statistically significant link between publishing the file and earning AI search citations.
Google Search Central's Stated Position
Google Search Central documentation explicitly confirms that Googlebot and Google AI Overviews do not consume /llms.txt for web discovery, content indexing, or search ranking (see Google Search Central Crawler Overview ). Google relies exclusively on HTML rendering, XML sitemaps, and standard Schema.org markup.
For detailed analysis of how this aligns with Chrome's Lighthouse audit, read our investigation into the llms.txt Google contradiction.
llms.txt vs robots.txt vs sitemap.xml
Comparing the technical scope, audience, and enforcement capabilities of web protocols.
| Feature / Protocol | llms.txt | robots.txt | sitemap.xml |
|---|---|---|---|
| Primary Purpose | Curated Markdown context for AI agents & LLMs | Crawler access control & crawl budget protection | Exhaustive URL inventory & canonical discovery |
| Controls Crawler Access? | No (informational only; no enforcement) | Yes (User-agent Allow / Disallow directives) | No (purely advisory discovery list) |
| Contains Page Descriptions? | Yes (Markdown descriptions per URL) | No | No (only technical metadata like lastmod) |
| Checked by Google Lighthouse? | Yes (Lighthouse 13.3+ Agentic Browsing audit) | Yes (validity and block tests) | No direct Lighthouse audit |
| Has v2 Discovery Link Relations? | Yes (rel="describedby", rel="alternate") | No | Partial (Sitemap pointer in robots.txt) |
| Google Search Indexing Support | No (Google ignores llms.txt for ranking) | Yes (Official Web Standard since 1994) | Yes (Official Sitemaps.org standard) |
| Target Audience | AI agents, LLM context windows, coding tools | Search engine bots, scrapers, web spiders | Search engine indexing crawlers (Googlebot, Bingbot) |
| File Format | Markdown (.txt) | Plain text key-value (.txt) | XML (.xml) |
How Our Generator Finds and Maps Your Pages
Standard scrapers fail on modern websites built with client-side JavaScript frameworks. Our engine uses an intelligent, progressive discovery chain to map your website accurately:
Robots.txt Sitemap Pointer Discovery
We inspect your domain's /robots.txt to extract declared sitemap directives and detect crawler policies.
Sitemap.xml & Index Parsing
We parse standard XML sitemaps, sitemap indexes, and timestamp metadata to locate your canonical, high-value pages.
JSON Sitemaps & RSS Feeds Fallback
If XML sitemaps are missing, we automatically probe for JSON feeds and RSS/Atom streams to discover fresh content.
Playwright Headless Browser JavaScript Rendering
For Single Page Applications (Next.js, React, Vue, Nuxt) where internal links are injected into the DOM via JavaScript, we spin up a headless Playwright instance to evaluate the DOM after client hydration.
Framework Detection & Path Filtering
We identify CMS footprints (WordPress, Shopify, Next.js) and automatically exclude non-content routes (such as /wp-admin/, /_next/, checkout, and auth endpoints).
How to Deploy Your llms.txt File
An llms.txt file must be accessible at the exact root of your domain (https://yourdomain.com/llms.txt). Here is how to deploy it on major platforms:
Next.js / Nuxt / Astro
Save the generated file to your project's public/llms.txt folder. The framework will automatically serve it statically at the domain root.
WordPress
Upload the file via SFTP or cPanel File Manager directly to the web root directory (/public_html/llms.txt). Do not put it inside /wp-content/.
Shopify
Upload llms.txt to Settings > Files in Shopify Admin. Then create a URL redirect under Online Store > Navigation > URL Redirects from /llms.txt to the uploaded file CDN URL.
Static Hosts (Vercel / Netlify / GitHub Pages)
Place the file in your static build output root (dist/llms.txt or static/llms.txt) so it is published at root on deployment.
How to Verify It Is Live
- Open a browser tab to
https://yourdomain.com/llms.txt. - Verify that the response returns HTTP status 200 OK and renders plain text Markdown.
- Confirm the
Content-Typeheader istext/plainortext/markdown; charset=utf-8. - Test syntax validity using our llms.txt Validator.
Frequently Asked Questions
Direct answers to common technical questions about llms.txt, AI search engine adoption, and Google Lighthouse auditing.
What is an llms.txt generator?▼
An llms.txt generator is a developer tool that crawls a website and formats its key URLs into a standardized Markdown file located at /llms.txt. It structures your site's core content, titles, and brief summaries so AI agents and large language models can efficiently discover and ingest authoritative pages without parsing complex HTML templates.
Is llms.txt an official standard?▼
No, llms.txt is an open community proposal maintained by llmstxt.org, not an official IETF or W3C standard. While not mandated by law or RFC specifications, it has gained substantial adoption among developer toolchains, AI documentation scrapers, and browser developer audits including Google Chrome's Lighthouse agentic browsing criteria.
Does Google use llms.txt?▼
No. Google Search Central has explicitly confirmed that Googlebot and Google AI Overviews do not read or use llms.txt for indexing, crawling, or search ranking. Google evaluates standard HTML, XML sitemaps, and structured schema data. However, Google Chrome's Lighthouse audit does inspect the file for agentic browsing readiness.
Will llms.txt get me cited by ChatGPT, Claude, or Perplexity?▼
There is no guarantee of citations. Primary research by Ahrefs and SE Ranking across more than 400,000 domains found zero statistical correlation between publishing an llms.txt file and receiving higher AI search citations. However, the file provides clean, structured Markdown context for developer tooling, autonomous coding agents, and direct user-prompted research assistants.
What does Lighthouse check in an llms.txt file?▼
Starting in Lighthouse 13.3 (May 2026), the Agentic Browsing audit verifies three core criteria if an /llms.txt file is detected: the file must begin with an H1 heading, contain valid Markdown links ([text](url)) rather than bare URLs, and exceed 50 characters in total length. Sites without an llms.txt file do not fail.
What changed in the llms.txt v2 specification?▼
The August 2026 v2 revision introduced rel="alternate" type="text/markdown" for discovering page-specific Markdown files, rel="describedby" to link HTML pages to their covering llms.txt file via HTML tags or HTTP headers, and subpath scoping (such as /docs/llms.txt), where the most specific file covers that directory. Both .md and .html.md extensions are supported.
How does llms.txt differ from robots.txt?▼
Robots.txt is an established protocol that controls crawler access and server resource usage using binary Allow and Disallow rules. It provides no content descriptions. In contrast, llms.txt does not enforce access control; it delivers a curated Markdown reading list with descriptive summaries specifically designed for AI comprehension and agent context windows.
How often should I regenerate my llms.txt file?▼
You should regenerate your llms.txt file whenever you publish major pillar content, restructure your core navigation, or deprecate key pages. For active blogs and SaaS applications, regenerating monthly or integrating automated generation into your CI/CD deployment pipeline ensures AI agents always receive up-to-date summaries and active URLs.
Primary Sources & Methodology
- Official llms.txt Specification (llmstxt.org) - Primary community standard for Markdown context files.
- llms.txt v2 Changelog (August 2026) - rel="alternate", rel="describedby", and subpath resolution.
- DebugBear: Lighthouse Agentic Browsing Audit - Audit thresholds in Chrome Lighthouse 13.3.
- Ahrefs llms.txt 137K Domain Study (May 2026) - Empirical bot request logs and adoption rates.
- Search Engine Journal: SE Ranking 300K Domain Analysis - AI citation statistical correlation report.
- Google Search Central Crawler Documentation - Official documentation on Googlebot and AI Overviews crawling.
Explore Related SEO & GEO Resources
llms.txt Validator
Test your existing file against Lighthouse Agentic Browsing audit criteria.
Generative Engine Optimization Guide
Master entity citation and content architecture for AI answer engines.
The Complete Guide to llms.txt
In-depth guide covering markdown hierarchy, token limits, and agent parsing.
llms.txt vs robots.txt
Understand why crawler permissions and semantic context require separate protocols.
How to Create an llms.txt File
Step-by-step tutorial on structuring, formatting, and hosting your file.
The llms.txt Google Contradiction
Why Google Search ignores the file while Chrome Lighthouse audits it.