# llms.txt

Generate an [llms.txt](https://llmstxt.org) file for a site: a Markdown index of its most important pages, with titles and descriptions, that language models and agents can read to find their way around. Optionally get `llms-full.txt` too, with the full Markdown of those pages in one file.

`POST https://xdive.dev/api/orgs/{id}/llmstxt`

Requires an API key with the `crawl` or the `scrape` scope. The request is synchronous and stops after 2 minutes.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/llmstxt" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "max_urls": 20, "full_text": true}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.llms_txt("https://docs.example.com", max_urls=20, full_text=True)

with open("llms.txt", "w") as f:
    f.write(result["llms_txt"])
with open("llms-full.txt", "w") as f:
    f.write(result["llms_full_txt"])
```

**JavaScript**

```ts
import { writeFile } from "node:fs/promises";
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true });

await writeFile("llms.txt", result.llms_txt as string);
await writeFile("llms-full.txt", result.llms_full_txt as string);
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The site's start page, an absolute `http` or `https` URL. It becomes the first entry, and its title and description head the file. |
| `max_urls` | integer | `10` | Pages to include, the start page among them. At most 20; larger values are capped. |
| `full_text` | boolean | `false` | Also return `llms_full_txt` with each page's full Markdown. |

## How pages are chosen

1. The site is mapped the way [Map](https://xdive.dev/docs/map.md) does it: its sitemaps plus a links-only crawl of up to 50 pages, 2 hops deep, within 30 seconds.
2. The start page comes first. The other URLs are ordered by path depth, shallowest first, so section landing pages come before deep articles. URLs ending in `.xml` or `.json`, and paths containing `login` or `wp-admin`, are skipped.
3. The chosen pages are scraped as main-content Markdown, four at a time, through the page cache and with robots.txt respected. Pages that fail are left out of the files.

## Response

```json
{
  "success": true,
  "llms_txt": "# Example Docs\n\n> Guides and API reference for Example.\n\n## Pages\n\n- [Example Docs](https://docs.example.com): Guides and API reference for Example.\n- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.\n- [Authentication](https://docs.example.com/authentication): API keys and scopes.\n",
  "pages": 3,
  "llms_full_txt": "# Example Docs\nSource: https://docs.example.com\n\n...\n\n---\n\n# Quickstart\nSource: https://docs.example.com/quickstart\n\n...\n\n---\n\n"
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true`. |
| `llms_txt` | string | The index file. |
| `pages` | integer | Pages scraped and included. |
| `llms_full_txt` | string | With `full_text`: every included page's Markdown, at most 500,000 bytes. |

The `llms_txt` file has this shape:

```markdown
# Example Docs

> Guides and API reference for Example.

## Pages

- [Example Docs](https://docs.example.com): Guides and API reference for Example.
- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.
- [Authentication](https://docs.example.com/authentication): API keys and scopes.
```

The heading is the start page's title (the host name when it has none or could not be scraped), followed by its meta description as a quote. Each entry links a page by its title, followed by its meta description when it has one. In `llms_full_txt`, each page starts with its title as a heading and a `Source:` line, and pages are separated by `---`. Pages are added until the file reaches 500,000 bytes, and the file is cut at that size.

## Usage

Each page scraped counts as one page against your monthly page quota. The request is refused with `402` when the quota is already used up.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `url` is missing or not an absolute `http` or `https` URL. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `crawl` nor the `scrape` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `502` | | No page could be scraped. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Map](https://xdive.dev/docs/map.md) for the full list of a site's URLs.
- [Crawl](https://xdive.dev/docs/crawl.md) to convert more than 20 pages.
- [MCP](https://xdive.dev/docs/mcp.md) to give an assistant live access to sites instead of a static file.

---

Source: https://xdive.dev/docs/llms-txt · All docs for agents: https://xdive.dev/llms.txt