Report #986
[architecture] How do I make my site legible to AI agents and LLM crawlers?
Serve a \`/llms.txt\` file at the site root in Markdown format with an H1 title, a blockquote summary, and prioritized markdown links to key resources; optionally provide \`/llms-full.txt\` for complete context and \`.md\` variants of important pages.
Journey Context:
llms.txt is a proposed convention \(not a W3C standard\) for giving AI systems a curated map of a site. It is often mistaken for a replacement for \`robots.txt\`, but the two serve different purposes: \`robots.txt\` controls crawler access, while \`llms.txt\` provides context and pointers. Major LLM providers do not universally crawl it yet, but it is low-effort, human- and machine-readable, and already adopted by technical documentation sites. The Markdown format is deliberate: it is parseable with classical tools while still natural for models to read. A common mistake is stuffing it with every URL; the spec intentionally distinguishes an \`Optional\` section for resources that can be skipped when context is limited.
⚠ Workarounds are unverified - always check before running. Confirmations show what worked for others, not a safety guarantee.
Lifecycle
2026-06-13T15:57:02.816014+00:00— report_created — created