llms.txt

Generate an llms.txt file for a site: a Markdown index of its most important pages, with titles and descriptions, that language models and agents can read to find their way around. Optionally get llms-full.txt too, with the full Markdown of those pages in one file.

Requires an API key with the crawl or the scrape scope. The request is synchronous and stops after 2 minutes.

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/llmstxt" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "max_urls": 20, "full_text": true}'

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.llms_txt("https://docs.example.com", max_urls=20, full_text=True)
 
with open("llms.txt", "w") as f:
    f.write(result["llms_txt"])
with open("llms-full.txt", "w") as f:
    f.write(result["llms_full_txt"])

import { writeFile } from "node:fs/promises";
import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true });
 
await writeFile("llms.txt", result.llms_txt as string);
await writeFile("llms-full.txt", result.llms_full_txt as string);

Parameters

How pages are chosen

1. The site is mapped the way Map does it: its sitemaps plus a links-only crawl of up to 50 pages, 2 hops deep, within 30 seconds.
2. The start page comes first. The other URLs are ordered by path depth, shallowest first, so section landing pages come before deep articles. URLs ending in .xml or .json, and paths containing login or wp-admin, are skipped.
3. The chosen pages are scraped as main-content Markdown, four at a time, through the page cache and with robots.txt respected. Pages that fail are left out of the files.

Response

{
  "success": true,
  "llms_txt": "# Example Docs\n\n> Guides and API reference for Example.\n\n## Pages\n\n- [Example Docs](https://docs.example.com): Guides and API reference for Example.\n- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.\n- [Authentication](https://docs.example.com/authentication): API keys and scopes.\n",
  "pages": 3,
  "llms_full_txt": "# Example Docs\nSource: https://docs.example.com\n\n...\n\n---\n\n# Quickstart\nSource: https://docs.example.com/quickstart\n\n...\n\n---\n\n"
}

The llms_txt file has this shape:

# Example Docs
 
> Guides and API reference for Example.
 
## Pages
 
- [Example Docs](https://docs.example.com): Guides and API reference for Example.
- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.
- [Authentication](https://docs.example.com/authentication): API keys and scopes.

The heading is the start page's title (the host name when it has none or could not be scraped), followed by its meta description as a quote. Each entry links a page by its title, followed by its meta description when it has one. In llms_full_txt, each page starts with its title as a heading and a Source: line, and pages are separated by ---. Pages are added until the file reaches 500,000 bytes, and the file is cut at that size.

Usage

Each page scraped counts as one page against your monthly page quota. The request is refused with 402 when the quota is already used up.

Errors

See Errors and limits for authentication errors and retries.

Related

• Map for the full list of a site's URLs.
• Crawl to convert more than 20 pages.
• MCP to give an assistant live access to sites instead of a static file.