# XDive documentation

> XDive is an HTTP API that turns web pages into data for code and AI agents: it scrapes one page as Markdown, HTML or structured JSON, crawls and maps whole sites, searches the web and news, extracts structured data with a prompt or JSON Schema, and answers questions with verified citations. It is available over REST (base URL https://xdive.dev/api, API key auth), through SDKs for Python, JavaScript, Go, Java, .NET, PHP, Ruby and Rust, and as a hosted MCP server at https://xdive.dev/api/mcp (OAuth or API key).


---

# Introduction

XDive is an HTTP API that turns web pages into data your code and AI agents can use. It fetches a page as Markdown, HTML or structured JSON, crawls whole sites, searches the web, and answers questions with cited sources. Use it directly over HTTP or through the official SDKs for Python, JavaScript, Go, Java, .NET, PHP, Ruby and Rust.

## What the API does

| Capability | Endpoint | What it does |
|---|---|---|
| [Scrape](https://xdive.dev/docs/scrape.md) | `POST /orgs/{id}/scrape` | One page as Markdown, HTML, links, text, a screenshot or JSON, returned in the same request. |
| [Crawl](https://xdive.dev/docs/crawl.md) | `POST /orgs/{id}/crawl` | Follows a site's links in the background; poll for status or receive webhooks. |
| [Batch scrape](https://xdive.dev/docs/batch-scrape.md) | `POST /orgs/{id}/batch/scrape` | Up to 100 URLs as one background job. |
| [Map](https://xdive.dev/docs/map.md) | `POST /orgs/{id}/map` | A site's URLs from its sitemaps and links, without fetching content. |
| [Search](https://xdive.dev/docs/search.md) | `POST /orgs/{id}/search` | Web, news and image search, optionally with page content and a cited answer. |
| [Research](https://xdive.dev/docs/research.md) | `POST /orgs/{id}/research` | Answers a question from the web with verified citations. |
| [Extract](https://xdive.dev/docs/extract.md) | `POST /orgs/{id}/extract` | Structured data from up to 10 URLs, driven by a prompt or a JSON Schema. |
| [llms.txt](https://xdive.dev/docs/llms-txt.md) | `POST /orgs/{id}/llmstxt` | Generates `llms.txt` and `llms-full.txt` for a site. |
| [MCP](https://xdive.dev/docs/mcp.md) | `POST /mcp` | The same tools for AI assistants such as Claude, ChatGPT and Cursor. |

## Base URL

```text
https://xdive.dev/api
```

A self-hosted server uses its own origin, for example `https://your-host/api` (see [Self-hosting](https://xdive.dev/docs/self-hosting.md)). Requests and responses are JSON; send `Content-Type: application/json` with a request body. The API is described in OpenAPI 3.1 at [`/openapi.json`](https://xdive.dev/openapi.json), and these docs are available to AI agents as Markdown through [`/llms.txt`](https://xdive.dev/llms.txt) (see [For AI agents](https://xdive.dev/docs/agents.md)).

## How requests are organized

Every account belongs to one or more organizations, and usage, quotas and API keys belong to an organization. Endpoints that do work are scoped to one, so their paths start with `/orgs/{id}/`, where `{id}` looks like `org_8f2c41d09a7b4e6f93d0c5a1b2e47f18`.

An API key belongs to exactly one organization. `GET /whoami` tells you which one, so you only need to configure the key: the SDKs call `/whoami` once, cache the `orgId` and build every path from it. A key sent to another organization's path is rejected with `401`. See [Authentication](https://xdive.dev/docs/authentication.md).

Scrape, map, search, research, extract and llms.txt answer within the request. Crawl and batch scrape return `202 Accepted` with a job id at once and run on background workers.

## A first request

Create a key in the dashboard under **API Keys** (the [Quickstart](https://xdive.dev/docs/quickstart.md) walks through it), export it as `XDIVE_API_KEY`, then scrape a page:

**cURL**

```bash
# Look up the key's organization once (uses jq), then scrape.
ORG_ID=$(curl -s https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY" | jq -r .orgId)

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape("https://example.com")
print(page["artifacts"]["markdown"]["content"])
```

**JavaScript**

```ts
import { XDive, pageMarkdown } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com");
console.log(pageMarkdown(page));
```

## Pick your path

- **New to the API:** the [Quickstart](https://xdive.dev/docs/quickstart.md) goes from sign-up to a first scrape, search and research call.
- **Using an SDK:** [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md), [Go](https://xdive.dev/docs/sdk-go.md), [Java](https://xdive.dev/docs/sdk-java.md), [.NET](https://xdive.dev/docs/sdk-dotnet.md), [PHP](https://xdive.dev/docs/sdk-php.md), [Ruby](https://xdive.dev/docs/sdk-ruby.md) or [Rust](https://xdive.dev/docs/sdk-rust.md), and the [Vite plugin](https://xdive.dev/docs/vite-plugin.md).
- **One page or a whole site:** [Scrape](https://xdive.dev/docs/scrape.md), [Crawl](https://xdive.dev/docs/crawl.md), [Map](https://xdive.dev/docs/map.md) and [Batch scrape](https://xdive.dev/docs/batch-scrape.md).
- **Answers from the web:** [Search](https://xdive.dev/docs/search.md) and [Research](https://xdive.dev/docs/research.md).
- **Structured data:** [Extract](https://xdive.dev/docs/extract.md), or the `json` format on [Scrape](https://xdive.dev/docs/scrape.md).
- **AI assistants and agent frameworks:** [MCP](https://xdive.dev/docs/mcp.md), [LangChain](https://xdive.dev/docs/langchain.md) and [LlamaIndex](https://xdive.dev/docs/llamaindex.md).
- **Before going to production:** [Authentication](https://xdive.dev/docs/authentication.md), [Errors and limits](https://xdive.dev/docs/errors.md) and [Usage and billing](https://xdive.dev/docs/usage.md).
- **On your own servers:** [Self-hosting](https://xdive.dev/docs/self-hosting.md).

---

Source: https://xdive.dev/docs/introduction · All docs for agents: https://xdive.dev/llms.txt


---

# Quickstart

Go from a new account to your first scrape, search and research call. You need an account, an API key and either `curl`, Python 3.9+ or Node.js 18+.

## 1. Create an account

Sign up at [xdive.dev/register](https://xdive.dev/register). Signing up also creates your first organization, which owns your API keys, usage and plan. New organizations start on the `free` plan (see [Usage and billing](https://xdive.dev/docs/usage.md#plans)).

> **Note: Servers that review sign-ups**
>
> A server can be configured to review new accounts (`REGISTRATION_APPROVAL=required`). On such a server, signing up records an access request instead of signing you in, and you receive an email when an administrator approves it. Until then, signing in returns `403` with the code `ACCOUNT_PENDING_APPROVAL`.

## 2. Create an API key

1. In the dashboard, open **API Keys** and click **Issue new key**.
2. Give the key a name, for example `Production server`.
3. Keep the default **Expiry (days)** of 90, or enter `0` for a key that never expires. Optionally restrict **Allowed IPs / CIDRs**.
4. Choose **Scopes**. New keys start with `crawl` and `scrape`; also select `search`, which search and research need.
5. Click **Create** and copy the key. It starts with `ck_live_` and is shown only once.

Scopes and key restrictions are covered in [Authentication](https://xdive.dev/docs/authentication.md#scopes).

## 3. Set up your environment

Store the key in an environment variable. Both SDKs read `XDIVE_API_KEY` when you don't pass a key.

```bash
export XDIVE_API_KEY="ck_live_…"
```

Install an SDK if you are not using cURL:

**cURL**

```bash
# Nothing to install. The examples below also use jq to read JSON.
```

**Python**

```bash
pip install xdive
```

**JavaScript**

```bash
npm install xdive
```

## 4. Find your organization id

Organization endpoints live under `/orgs/{id}/`. `GET /whoami` returns the organization your key belongs to. The SDKs make this call for you on first use and cache the result, so only cURL users need to save the id.

**cURL**

```bash
curl https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY"

# Save the id for the next steps:
export ORG_ID=$(curl -s https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY" | jq -r .orgId)
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
print(xdive.whoami())
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
console.log(await xdive.whoami());
```

```json
{
  "auth": "api_key",
  "keyId": "key_5b1e9c7a20d34f8e86a1c0d7e3f92b64",
  "orgId": "org_8f2c41d09a7b4e6f93d0c5a1b2e47f18",
  "plan": "free",
  "scopes": ["crawl", "scrape", "search"],
  "suspended": false
}
```

## 5. Scrape a page

Fetch one page as Markdown. Scrape is synchronous: the content comes back in the response.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()
page = xdive.scrape("https://example.com")
print(page["title"])
print(page["artifacts"]["markdown"]["content"])
```

**JavaScript**

```ts
import { XDive, pageMarkdown } from "xdive";

const xdive = new XDive();
const page = await xdive.scrape("https://example.com");
console.log(page.title);
console.log(pageMarkdown(page));
```

The API wraps the page in `data`; the SDKs return `data` directly.

```json
{
  "data": {
    "url": "https://example.com",
    "title": "Example Domain",
    "depth": 0,
    "status_code": 200,
    "word_count": 19,
    "fetched_at": "2026-10-15T08:42:10Z",
    "fetch_route": "http",
    "fetch_attempts": 1,
    "fetch_cost_units": 1,
    "artifacts": {
      "markdown": {
        "media_type": "text/markdown; charset=utf-8",
        "content": "# Example Domain\n\nThis domain is for use in documentation examples …",
        "bytes": 167
      }
    }
  },
  "security": {
    "cache_state": "eligible",
    "network_policy": "live",
    "outbound_permitted": true,
    "zero_data_retention": false
  },
  "success": true
}
```

Ask for other formats with `formats`, for example `["markdown", "links"]`. See [Scrape](https://xdive.dev/docs/scrape.md) for every option.

## 6. Search the web

Search returns ranked results from the configured search engines. This call needs the `search` scope.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/search" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"query": "robots.txt specification", "limit": 3}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()
results = xdive.search("robots.txt specification", limit=3)
for result in results["results"]:
    print(result["rank"], result["title"], result["url"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive();
const { results } = await xdive.search("robots.txt specification", { limit: 3 });
for (const result of results) console.log(result.rank, result.title, result.url);
```

```json
{
  "provider": "brave",
  "query": "robots.txt specification",
  "results": [
    {
      "id": "3fa94c1e0b7d2a85",
      "source": "web",
      "rank": 1,
      "title": "RFC 9309: Robots Exclusion Protocol",
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
      "description": "The standard that defines how crawlers read robots.txt rules."
    }
  ],
  "search_depth": ""
}
```

Add `"include_answer": true` for a short cited answer, or `"search_depth": "advanced"` to also read the top results and get their most relevant passages. See [Search](https://xdive.dev/docs/search.md).

## 7. Ask a research question

Research searches the web, reads the best sources and writes an answer whose citations are checked against the pages it read. Quick research (the default) takes seconds; `"depth": "deep"` plans several searches and can take up to about two minutes. It needs the `search` scope.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/research" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"query": "What does RFC 9309 standardize?"}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()
result = xdive.research("What does RFC 9309 standardize?")
print(result["answer"])
for citation in result["citations"]:
    print(f'[{citation["number"]}] {citation["url"]}')
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive();
const result = await xdive.research("What does RFC 9309 standardize?");
console.log(result.answer);
for (const citation of result.citations) console.log(`[${citation.number}] ${citation.url}`);
```

```json
{
  "query": "What does RFC 9309 standardize?",
  "answer": "RFC 9309 standardizes the Robots Exclusion Protocol, the robots.txt rules that tell crawlers which paths of a site they may fetch [1]. …",
  "citations": [
    {
      "number": 1,
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
      "title": "RFC 9309: Robots Exclusion Protocol",
      "quote": "…"
    }
  ],
  "sources": [
    {
      "number": 1,
      "url": "https://www.rfc-editor.org/rfc/rfc9309.html",
      "title": "RFC 9309: Robots Exclusion Protocol",
      "passages": [{ "text": "…", "score": 7.42, "start": 0 }],
      "score": 7.42
    }
  ],
  "queries": ["What does RFC 9309 standardize?"],
  "confidence": "high",
  "usage": {
    "searches": 1,
    "pages_read": 4,
    "rounds": 1,
    "plan": "single",
    "duration_ms": 6120,
    "search_ms": 830,
    "read_ms": 2410,
    "model_ms": 2760
  }
}
```

> **Note:**
>
> Research writes its answer with a language model: the one your organization configured under **AI Extraction** in the dashboard, or one the server provides. Without either, the call returns `501`. Asking the same question with the same options again within 10 minutes can return the cached answer, marked `"cached": true`, and a cached answer is not metered.

## Next steps

- Crawl a whole site with [Crawl](https://xdive.dev/docs/crawl.md), or list its URLs first with [Map](https://xdive.dev/docs/map.md).
- Learn the client libraries: [Python SDK](https://xdive.dev/docs/sdk-python.md) and [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md).
- Handle limits and retries before going live: [Errors and limits](https://xdive.dev/docs/errors.md).
- Connect an AI assistant: [MCP](https://xdive.dev/docs/mcp.md).

---

Source: https://xdive.dev/docs/quickstart · All docs for agents: https://xdive.dev/llms.txt


---

# Authentication

Programs authenticate with an organization API key. The dashboard uses a session token for the signed-in user, and AI assistants that connect over MCP can sign in with OAuth instead of holding a key.

| Credential | Who uses it | Sent as | Access |
|---|---|---|---|
| API key (`ck_live_…`) | Your code, the SDKs, MCP clients that can send headers | `X-API-Key` or `Authorization: Bearer` | One organization, limited to the key's scopes |
| Session token (JWT) | The dashboard | `Authorization: Bearer` | Every organization the user belongs to, no scope limits |
| OAuth access token | MCP clients such as ChatGPT and claude.ai | `Authorization: Bearer` | One organization, MCP endpoint only |

## API keys

An API key belongs to exactly one organization. It looks like `ck_live_` followed by 32 lowercase letters and digits. Only a SHA-256 hash of the key is stored, so the full key is shown once, when it is created; afterwards the dashboard shows its first 12 characters (for example `ck_live_7hq2…`) and when it was last used.

### Sending the key

Send the key in either header. The SDKs use `X-API-Key`.

| Header | Example |
|---|---|
| `X-API-Key` | `X-API-Key: ck_live_…` |
| `Authorization` | `Authorization: Bearer ck_live_…` |

A `Bearer` value is treated as an API key only when it starts with `ck_`; anything else is treated as a session token. If both headers are present, `X-API-Key` is used.

**cURL**

```bash
curl https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY"
```

**Python**

```python
from xdive import XDive

xdive = XDive(api_key="ck_live_…")  # or set XDIVE_API_KEY
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive({ apiKey: "ck_live_…" }); // or set XDIVE_API_KEY
```

A key used under another organization's path (`/orgs/{id}/…` with a different `{id}`) is rejected with `401 invalid API key`.

### Creating and revoking keys

Create keys in the dashboard under **API Keys** → **Issue new key**. Keys can also be managed over the API with a [session token](#dashboard-sessions); an API key cannot create, list or revoke keys.

`POST https://xdive.dev/api/orgs/{id}/api-keys`

| Field | Type | Default | Description |
|---|---|---|---|
| `name` | string | `"Untitled key"` | A label shown in the dashboard. |
| `scopes` | string[] | `["crawl", "scrape"]` | What the key may do; see [Scopes](#scopes). |
| `expires_in_days` | integer | `0` | Days until the key stops working, from 1 to 3650. `0` means it never expires. The dashboard suggests 90. |
| `ip_allowlist` | string[] | `[]` | Up to 32 IP addresses or CIDR ranges the key may be used from. Empty allows any address. |

The response (`201 Created`) is the only one that includes `secret`:

```json
{
  "id": "key_5b1e9c7a20d34f8e86a1c0d7e3f92b64",
  "orgId": "org_8f2c41d09a7b4e6f93d0c5a1b2e47f18",
  "name": "Production server",
  "prefix": "ck_live_7hq2",
  "secret": "ck_live_7hq2…",
  "scopes": ["crawl", "scrape", "search"],
  "createdAt": "2026-10-15T08:30:00Z",
  "lastUsedAt": null,
  "expiresAt": "2027-01-13T08:30:00Z",
  "ipAllowlist": ["203.0.113.10", "198.51.100.0/24"]
}
```

`GET /orgs/{id}/api-keys` lists the organization's keys without secrets. `DELETE /orgs/{id}/api-keys/{keyId}` revokes a key and returns `204`; the key stops working immediately on every server.

### Scopes

Each key carries a list of scopes. A request outside them fails with `403` and a message such as `this API key lacks the 'search' scope`.

| Scope | Allows |
|---|---|
| `scrape` | `GET` and `POST /orgs/{id}/scrape` (including scrapes with browser actions or screenshots), `POST /orgs/{id}/parse` |
| `crawl` | `POST /orgs/{id}/crawl`, `POST /orgs/{id}/map` |
| `scrape` or `crawl` | `POST /orgs/{id}/batch/scrape`, `GET` and `DELETE /orgs/{id}/crawls/{crawlId}`, `GET /orgs/{id}/crawls/{crawlId}/errors`, `POST /orgs/{id}/extract`, `POST /orgs/{id}/llmstxt` |
| `search` | `POST /orgs/{id}/search`, `POST /orgs/{id}/research` |
| `browser` | Interactive browser sessions and saved browser profiles under `/orgs/{id}/browser/` |
| `connectors` | Data connectors under `/orgs/{id}/connectors` |

A new key may carry only these five scopes; any other value is rejected with `400`, which lists the valid ones. Batch scrape results are read through the crawl endpoints, which is why those accept `scrape` as well as `crawl`.

Organization management is not available to API keys at all: API keys, usage, plan and billing, schedules and monitors, history, AI settings, the audit log, crawl export and connected apps all require a session token.

### Expiry and IP allowlists

An expired key, or a key used from an address outside its allowlist, is rejected with `401 invalid API key`, the same answer as for an unknown key. The client address is the connection's address, or the `X-Forwarded-For` address when the request arrives through a proxy the server trusts.

### Suspended organizations

Keys of a suspended organization are rejected with `403 organization is suspended`. `GET /whoami` still answers and reports `"suspended": true`.

## Find the organization

`GET /whoami` identifies the caller. It accepts an API key or a session token, and the SDKs use it to discover the organization id from the key.

`GET https://xdive.dev/api/whoami`

**cURL**

```bash
curl https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY"
```

**Python**

```python
from xdive import XDive

who = XDive().whoami()
print(who["orgId"], who["scopes"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const who = await new XDive().whoami();
console.log(who.orgId, who.scopes);
```

With an API key:

```json
{
  "auth": "api_key",
  "keyId": "key_5b1e9c7a20d34f8e86a1c0d7e3f92b64",
  "orgId": "org_8f2c41d09a7b4e6f93d0c5a1b2e47f18",
  "plan": "free",
  "scopes": ["crawl", "scrape", "search"],
  "suspended": false
}
```

| Field | Type | Description |
|---|---|---|
| `auth` | string | `api_key` |
| `keyId` | string | The key's id (`key_…`). |
| `orgId` | string | The organization the key belongs to (`org_…`). |
| `scopes` | string[] | The key's scopes. |
| `plan` | string | The organization's plan: `free`, `pro` or `enterprise`. |
| `suspended` | boolean | Whether the organization is suspended. |

With a session token, `auth` is `user` and the response lists the user's organizations:

```json
{
  "auth": "user",
  "orgs": [
    {
      "id": "org_8f2c41d09a7b4e6f93d0c5a1b2e47f18",
      "name": "Acme",
      "plan": "free",
      "role": "owner",
      "createdAt": "2026-10-01T09:12:44Z"
    }
  ],
  "userId": "usr_0c7d52e1a94b4f3e8d6a2b19f50c3e7a"
}
```

An invalid key returns `401 invalid API key`; a missing or invalid session returns `401 unauthorized`.

## Dashboard sessions

Signing in (`POST /auth/login` with `email` and `password`) returns a session `token`: a JWT that is valid for 7 days and is sent as `Authorization: Bearer <token>`.

```bash
curl -X POST https://xdive.dev/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"email": "you@example.com", "password": "…"}'
```

A session acts as the user. It can use every organization the user is a member of and is not limited by scopes, which is why it is meant for the dashboard and the management endpoints rather than for programs. The account's status is checked on every request: if an administrator has not approved the account yet, or has rejected it, the session receives `403` with `ACCOUNT_PENDING_APPROVAL` or `ACCOUNT_REJECTED` (see [Errors and limits](https://xdive.dev/docs/errors.md#error-codes)).

## OAuth for AI assistants

MCP clients that cannot hold an API key, such as ChatGPT and claude.ai connectors, sign in with OAuth 2.1. The client discovers the server from `/.well-known/oauth-protected-resource` and `/.well-known/oauth-authorization-server`, registers itself, and sends you to a consent screen where you choose the organization. The flow uses the authorization code grant with PKCE (`S256` only).

- Access tokens last 1 hour; refresh tokens last 30 days and are rotated on use.
- Tokens are bound to the MCP endpoint (`/api/mcp`). The REST API rejects them, so they cannot be used for direct API calls.
- A grant can scrape, crawl and search for the organization you chose.
- Review and revoke connected clients under **API Keys** → **Connected apps**.

ChatGPT, Claude, Claude Code, Codex, Cursor, VS Code, Gemini CLI and Windsurf all sign in this way. Clients and scripts that cannot run a browser sign-in can send an API key in a header instead. See [MCP](https://xdive.dev/docs/mcp.md) for setup.

## Device sign-in for CLIs and agents

A command-line tool or AI agent can get an API key with one approval in the browser, using the OAuth 2.0 Device Authorization Grant ([RFC 8628](https://www.rfc-editor.org/rfc/rfc8628)). The agent shows a short code, you open `/device`, check the code, choose the organization and click **Allow**, and the agent receives a normal organization API key. Nothing has to be copied or pasted.

The result is an API key rather than an OAuth access token on purpose: it works with every REST endpoint and with MCP, carries scopes like any other key, never needs refreshing, and is listed under **API Keys** (as `Device: <client name> <date>`), where you can revoke it. The server advertises the endpoint as `device_authorization_endpoint` in `/.well-known/oauth-authorization-server`.

**1. Ask for a code.** `client_id` is optional; without it the default public client `cli` is used, and `client_name` sets the name you will see (it is shown as unverified). `scope` is a space-separated list of [key scopes](#scopes) and defaults to `scrape crawl search`.

`POST https://xdive.dev/api/oauth/device_authorization`

```bash
curl -X POST https://xdive.dev/api/oauth/device_authorization \
  -d client_name="My agent" \
  -d scope="scrape crawl search"
```

```json
{
  "device_code": "dc_Qm9vdGgtY29kZS1leGFtcGxlLW5vdC1yZWFs",
  "user_code": "WDJB-MJHT",
  "verification_uri": "https://xdive.dev/device",
  "verification_uri_complete": "https://xdive.dev/device?user_code=WDJB-MJHT",
  "expires_in": 600,
  "interval": 5,
  "scope": "scrape crawl search"
}
```

**2. Send the person to the page.** Show `user_code` and `verification_uri`, or open `verification_uri_complete`, which fills in the code. They sign in if needed, compare the code, pick an organization and approve or deny. The code works for 10 minutes and only once; typing it in lower case or without the dash is fine.

**3. Poll for the key.** Every `interval` seconds, until you get a key or a final error:

```bash
curl -X POST https://xdive.dev/api/oauth/token \
  -d grant_type=urn:ietf:params:oauth:grant-type:device_code \
  -d device_code="$DEVICE_CODE"
```

If you passed a `client_id` in step 1, send the same one here. Once approved, the response carries the key in `access_token`. It is returned exactly once:

```json
{
  "access_token": "ck_live_…",
  "token_type": "Bearer",
  "scope": "scrape crawl search",
  "org_id": "org_8f2c41d09a7b4e6f93d0c5a1b2e47f18",
  "key_id": "key_5b1e9c7a20d34f8e86a1c0d7e3f92b64",
  "key_name": "Device: My agent 2026-10-17"
}
```

Until then the endpoint answers `400` with one of these `error` values:

| `error` | Meaning | What to do |
|---|---|---|
| `authorization_pending` | Not approved yet. | Keep polling at `interval`. |
| `slow_down` | You polled too fast. | Add 5 seconds to your interval, then keep polling. |
| `access_denied` | The person clicked **Deny**, or the account or organization lost access. | Stop. |
| `expired_token` | 10 minutes passed. | Start again from step 1. |
| `invalid_grant` | Unknown code, another client's code, or a key was already issued for it. | Stop. |

A suspended organization, or an account waiting for approval or rejected, gets no key (`403`). Requests for codes are rate-limited per IP address, and the `/device` page locks a user out for 15 minutes after 10 wrong codes.

Use the key like any other: `Authorization: Bearer ck_live_…` against `/api/orgs/{org_id}/…`.

## Keeping keys secret

- Use keys from server-side code only. Never put a key in browser or mobile code, or commit it to a repository. Browser requests from origins the server does not allow are refused with `CORS_ORIGIN_DENIED`.
- Load keys from the environment (`XDIVE_API_KEY`) or a secret manager.
- Create one key per application and environment with only the scopes it needs, an expiry, and an IP allowlist where your servers have fixed addresses.
- Rotate by issuing a new key, deploying it, then revoking the old one.
- If a key leaks, revoke it in **API Keys**. Revocation takes effect immediately.

---

Source: https://xdive.dev/docs/authentication · All docs for agents: https://xdive.dev/llms.txt


---

# For AI agents

This page is for AI agents and the people who build them: which endpoint or MCP tool fits which job, how to authenticate, what things cost, and how to retry safely. Every docs page is also available as plain Markdown, so an agent can read the docs without rendering this site.

## Machine-readable docs

| Resource | URL | Contents |
|---|---|---|
| `llms.txt` | [https://xdive.dev/llms.txt](https://xdive.dev/llms.txt) | An index of every docs page in the [llmstxt.org](https://llmstxt.org) format, linking to the Markdown versions. |
| `llms-full.txt` | [https://xdive.dev/llms-full.txt](https://xdive.dev/llms-full.txt) | All docs pages in one Markdown file. |
| Markdown pages | `https://xdive.dev/docs/<slug>.md` | Each page as Markdown, for example [scrape.md](https://xdive.dev/docs/scrape.md). Docs pages advertise theirs with `<link rel="alternate" type="text/markdown">` and a `Link` header. |
| OpenAPI | [https://xdive.dev/openapi.json](https://xdive.dev/openapi.json) | An OpenAPI 3.1 description of the public REST API: every endpoint, its parameters, responses, scopes and error codes. Generate a client or tool definitions from it. |
| MCP server | `https://xdive.dev/api/mcp` | The hosted [MCP](https://xdive.dev/docs/mcp.md) server (Streamable HTTP). |

## Choose the endpoint or tool

Pick the cheapest call that answers the need. MCP tools call the same API, so the same limits and metering apply.

| Job | REST endpoint | MCP tool |
|---|---|---|
| Find pages about a topic | `POST /orgs/{id}/search` | `search` |
| Recent news | `POST /orgs/{id}/search` with `sources: ["news"]` and `freshness` | `search` with `sources: ["news"]` |
| Read one known URL as Markdown | `POST /orgs/{id}/scrape` | `fetch` (or `scrape` for other formats) |
| A page as HTML, links, a screenshot or structured JSON | `POST /orgs/{id}/scrape` with `formats` | `scrape` with `formats` |
| A cited answer to a question | `POST /orgs/{id}/research` | `research` |
| Fields or tables from up to 10 pages | `POST /orgs/{id}/extract` | `extract` |
| Find the right URL on a site before reading it | `POST /orgs/{id}/map` with `search` | `map` |
| Read many pages of one site | `POST /orgs/{id}/crawl`, then `GET /orgs/{id}/crawls/{crawlId}` | `crawl`, then `crawl_status` |
| A known list of up to 100 URLs | `POST /orgs/{id}/batch/scrape` | (none; use `fetch` per URL) |
| An `llms.txt` for someone else's site | `POST /orgs/{id}/llmstxt` | (none) |
| Convert an uploaded PDF, DOCX or XLSX | `POST /orgs/{id}/parse` (see the OpenAPI spec) | (none) |
| Click, type and scroll in a real browser | `POST /orgs/{id}/scrape` with [`actions`](https://xdive.dev/docs/scrape.md#browser-actions); interactive sessions under `/orgs/{id}/browser/sessions` (see the OpenAPI spec) | (none) |

Rules of thumb:

- Prefer search then scrape (or `fetch`) when you need to quote precisely; prefer research when the user wants an answer, not documents.
- Do not crawl a site to read one page: map it with `search`, then scrape the page you need.
- Keep crawls small and targeted with `max_pages`, `max_depth` and `include_paths`.
- Crawl and batch scrape answer `202` with a job id. Poll `GET /orgs/{id}/crawls/{crawlId}`, honouring `Retry-After` and `ETag` ([details](https://xdive.dev/docs/crawl.md#polling-with-etag-and-retry-after)), or pass `webhook_url`.

## Authentication

There are three ways for an agent to authenticate. See [Authentication](https://xdive.dev/docs/authentication.md) for the details.

- **API key** (REST and MCP). Send `Authorization: Bearer ck_live_…` or `X-API-Key: ck_live_…`. A key belongs to one organization; call `GET /whoami` once to get its `orgId` and build `/orgs/{orgId}/…` paths from it. Keys carry scopes (`scrape`, `crawl`, `search`, `browser`, `connectors`); new keys get `scrape` and `crawl`, so add `search` for search and research. Keep keys in environment variables, never in prompts, code or logs.
- **Device sign-in** (REST and MCP), for an agent or CLI without a key. Call `POST /api/oauth/device_authorization`, show the person the `verification_uri` and the `user_code`, and poll `POST /api/oauth/token` every `interval` seconds. Once the person approves at `/device`, the token response carries an API key and its `org_id`. See [Device sign-in](https://xdive.dev/docs/authentication.md#device-sign-in-for-clis-and-agents).
- **OAuth** (MCP only). MCP clients such as Claude, ChatGPT, Cursor and Codex sign in with OAuth 2.1 and PKCE: paste the server URL, sign in, choose the organization, approve. No key is copied. OAuth tokens work only on `/api/mcp`, not on the REST API.

API keys cannot manage the organization: API keys, usage, schedules, history and exports need a dashboard session.

## Costs and limits

Usage is metered per organization and month in pages, searches and browser seconds. Map, status polling, cancellation and `GET /whoami` are free, and so are requests that are rejected before any work starts. See [Usage and billing](https://xdive.dev/docs/usage.md) for what each endpoint counts, and [Errors and limits](https://xdive.dev/docs/errors.md#quotas) for the monthly quotas. A used-up quota answers `402`; that is something to tell the user, not to retry.

If a person gives you an API key, it may have its own monthly page and search budget, set so an agent cannot spend the whole organization's quota. Over that budget the API also answers `402` `QUOTA_EXCEEDED`, and the message names the key's budget. See [Per-key budgets](https://xdive.dev/docs/usage.md#per-key-budgets).

Synchronous calls (scrape, map, search, research, extract, llms.txt, parse) time out on the server: about 30 seconds for a scrape, longer for deep research. Set client timeouts to at least 120 seconds, and 180 seconds for `depth: "deep"` research.

## Rate limits

Each limit answers `429` (or `503` when a server is at capacity) with a `Retry-After` header in seconds. See [Rate limits and concurrency](https://xdive.dev/docs/errors.md#rate-limits-and-concurrency) for the numbers per plan.

Work-creating requests also report the per-minute budget in standard headers:

| Header | Meaning |
|---|---|
| `RateLimit-Limit` | Requests allowed in the current window. |
| `RateLimit-Remaining` | Requests left in the current window. |
| `RateLimit-Reset` | Seconds until the window resets. |

The older `X-RateLimit-Limit` and `X-RateLimit-Remaining` headers carry the same numbers. Slow down when `RateLimit-Remaining` gets close to zero instead of waiting for a `429`.

## Retries and idempotency

- **`429` and `503`:** wait `Retry-After` seconds (1 when it is missing), then retry. Cap the number of retries.
- **`502` and `504`:** a dependency failed or was slow. One retry may succeed, and is metered like any request.
- **`402`, `401`, `403` and other `4xx`:** do not retry the same request. Fix it or report it.
- Every error body has `retryable`: retry only when it is `true`. Branch on the HTTP status and the error body's `code` (for example `CONCURRENCY_LIMIT`), never on the `error` text. Include the `X-Request-ID` response header when you report a problem.

To retry a `POST` safely, send an `Idempotency-Key` header with a unique value (a UUID) per logical request, and reuse it on every retry of that request:

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: 7f0c2a9e-3b1d-4c55-9a7e-2d4f6b8c1e03" \
  -d '{"url": "https://docs.example.com", "max_pages": 50}'
```

- The same key with the same body within 24 hours returns the stored response, with the header `Idempotent-Replayed: true`, instead of doing the work (and metering it) again. A retried crawl therefore does not start a second crawl.
- The same key with a different body is rejected with `422` and the code `IDEMPOTENCY_KEY_REUSED`. Use a new key for a new request.

## Instructions for your agent

Drop-in instruction files that teach a coding agent to use the API correctly:

| File | For | Install |
|---|---|---|
| [AGENTS.md](https://xdive.dev/agents/AGENTS.md) | Codex, Cursor, Copilot, Gemini CLI and other agents that read `AGENTS.md` | Append it to the `AGENTS.md` at the root of your repository. |
| [SKILL.md](https://xdive.dev/agents/xdive-api/SKILL.md) | Claude Code and the Claude Agent SDK | Save it as `.claude/skills/xdive-api/SKILL.md` in your repository (or `~/.claude/skills/xdive-api/SKILL.md`). |
| [xdive.mdc](https://xdive.dev/agents/xdive.mdc) | Cursor | Save it as `.cursor/rules/xdive.mdc`. |

```bash
curl -fsSL https://xdive.dev/agents/AGENTS.md >> AGENTS.md
mkdir -p .claude/skills/xdive-api && curl -fsSL -o .claude/skills/xdive-api/SKILL.md https://xdive.dev/agents/xdive-api/SKILL.md
mkdir -p .cursor/rules && curl -fsSL -o .cursor/rules/xdive.mdc https://xdive.dev/agents/xdive.mdc
```

For assistants that use the tools directly rather than writing code against the API, connect the [MCP server](https://xdive.dev/docs/mcp.md); its plugin for Claude Code and Codex adds a `web-research` skill on top of the same tools.

---

Source: https://xdive.dev/docs/agents · All docs for agents: https://xdive.dev/llms.txt


---

# Command line

`xdive` is the API from a shell. It is part of the `xdive` npm package, so it runs with `npx` and needs nothing installed beyond Node.js 18. It is built for coding agents such as Codex and Claude Code as much as for people: data goes to stdout, every message to stderr, and the exit code says what happened.

```bash
npx xdive login
npx xdive scrape https://docs.stripe.com/payments
```

## Sign in

`xdive login` uses [device sign-in](https://xdive.dev/docs/authentication.md#device-sign-in-for-clis-and-agents): it prints a link and a code, you approve once in the browser, and the CLI saves a new API key with the `scrape`, `crawl` and `search` scopes to `~/.config/xdive/config.json` (readable only by you). The key is listed under **API Keys**, where you can revoke it. `xdive logout` deletes the saved key from this machine.

In CI or an agent sandbox, set `XDIVE_API_KEY` instead; it takes precedence over the saved key. `XDIVE_BASE_URL` or `--base-url` points the CLI at a self-hosted server.

## Commands

| Command | Does |
|---|---|
| `xdive scrape <url>` | A page as Markdown. `--max-chars` and `--offset` read a long page in parts, `--wait-for <ms>` lets a slow page render, `--format markdown,links,html` asks for other formats (printed as JSON). |
| `xdive search <query>` | Search results: title, URL and snippet. `--limit` sets how many. |
| `xdive research <question>` | A cited answer with its sources. `--deep` researches longer. |
| `xdive map <url>` | A site's URLs, one per line. `--search` keeps those about a topic. |
| `xdive crawl <url>` | Several pages of a site as Markdown, waiting for the crawl to finish. `--max-pages` (default 10) and `--max-depth` (default 2). |
| `xdive extract <url...> --prompt "…"` | Structured data from up to 10 pages, as JSON. |
| `xdive whoami` | The organization and key in use. |

Add `--json` to any command for the API's JSON response.

## For agents

- **Output.** stdout carries only the data: Markdown by default, JSON with `--json`. Warnings (for example `content may be incomplete`) and the "continue with `--offset N`" hint for long pages go to stderr, so `xdive scrape URL > page.md` saves just the page.
- **Long pages.** `xdive scrape URL --max-chars 8000` returns the first part; stderr then names the next offset. The parts come from one cached fetch.
- **Exit codes.**

| Code | Meaning | What to do |
|---|---|---|
| `0` | Success | |
| `1` | Other error (the message is on stderr) | Read the message; do not loop. |
| `2` | Wrong usage | Fix the command; `xdive --help` lists the options. |
| `3` | Not signed in, or the key was refused | Run `xdive login` or set `XDIVE_API_KEY`. |
| `4` | Quota or key budget used up | Tell the user; retrying will not help. |
| `5` | Rate limited | Wait the number of seconds on stderr, then retry. |

The CLI already retries rate limits and network failures a few times with an `Idempotency-Key`, so a retry never runs or bills the work twice.

---

Source: https://xdive.dev/docs/cli · All docs for agents: https://xdive.dev/llms.txt


---

# Python SDK

The `xdive` package wraps the API in two clients: `XDive` (synchronous) and `AsyncXDive` (asyncio). Each method makes one API call, or polls until a job finishes, and returns the parsed JSON. The clients discover your organization from the API key, retry rate-limited requests, and raise typed exceptions.

## Install

```bash
pip install xdive
```

Python 3.9 or later. The only runtime dependency is `httpx`. The package ships type annotations and a `py.typed` marker, so type checkers use them.

Optional extras install the framework integrations:

| Extra | Installs | Adds |
|---|---|---|
| `xdive[langchain]` | `langchain-core` | `xdive.langchain`: agent tools and a document loader. See [LangChain](https://xdive.dev/docs/langchain.md). |
| `xdive[llama-index]` | `llama-index-core` | `xdive.llama_index`: `XDiveReader`. See [LlamaIndex](https://xdive.dev/docs/llamaindex.md). |

```bash
pip install "xdive[langchain]"
pip install "xdive[llama-index]"
```

## Configure the client

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY and, if set, XDIVE_BASE_URL

xdive = XDive(
    api_key="ck_live_…",
    base_url="https://xdive.dev/api",
    org_id="org_3f9a1c2e7b5d4a6f8e0c1b2a3d4e5f60",
    timeout=60.0,
    max_retries=5,
)
```

| Argument | Environment variable | Default | Description |
|---|---|---|---|
| `api_key` | `XDIVE_API_KEY` | required | Organization API key (`ck_live_…`). The argument wins over the variable. Creating a client without either raises `ValueError`. |
| `base_url` | `XDIVE_BASE_URL` | `https://xdive.dev/api` | API base URL, including `/api`. A trailing slash is removed. |
| `org_id` | none | discovered | Organization id. Pass it to skip the `/whoami` lookup. |
| `timeout` | none | `120.0` | Per-request timeout in seconds, applied to every HTTP call. |
| `max_retries` | none | `3` | How many times an overload response (`429`, or `503` with code `SERVER_BUSY`) is retried. See [Retries](#retries-and-rate-limits). |

`api_key` is the only positional argument; the others are keyword-only. Every request sends the key in the `X-API-Key` header.

An API key belongs to one organization, and organization endpoints live under `/orgs/{id}/`. The first call that needs the organization id calls `GET /whoami` and caches the `orgId` it returns for the life of the client. `crawler.org_id` is a property that triggers the same lookup.

The key's scopes decide which methods work: `scrape` for scraping, `crawl` for crawls and maps, `search` for search and research. A call outside the key's scopes raises `XDiveError` with status `403`. See [Authentication](https://xdive.dev/docs/authentication.md).

## A first request

```python
from xdive import XDive, page_markdown

with XDive() as xdive:
    page = xdive.scrape("https://example.com")
    print(page["title"])
    print(page_markdown(page))
```

`scrape` returns the page object from the response's `data` field. Each requested format is an entry under `artifacts`, keyed by the format's name; Markdown is the default format. `page_markdown(page)` returns the Markdown whether it is in `artifacts["markdown"]["content"]` (a scrape, or a crawl with `formats`) or in `markdown` (a crawl without `formats`), and an empty string when there is none.

## Methods

Keyword options are sent as the API's JSON fields, unchanged, so every field in the [API reference](https://xdive.dev/docs/scrape.md) is available even when the SDK does not name it. Options set to `None` are left out of the request.

`formats` takes the API's format names: `markdown`, `text`, `html`, `rawHtml`, `links`, `images`, `json`, `summary`, `question`, `highlights`, `branding`, `audio`, `screenshot`, `product` and `menu` (`audio` and `screenshot` on `scrape` only). A name is sent as `{"type": "markdown"}`; pass a dict, such as `{"type": "json", "schema": {...}}` or `{"type": "question", "question": "..."}`, to set a format's options.

| Method | API call | Returns |
|---|---|---|
| `whoami()` | `GET /whoami` | `dict`: the key's identity. |
| `scrape(url, formats=None, **options)` | `POST /orgs/{id}/scrape` | `dict`: the page (`data`). |
| `crawl(url, *, timeout=600.0, poll_interval=2.0, **options)` | `POST /orgs/{id}/crawl`, then polls | `dict`: the finished crawl record. |
| `start_crawl(url, **options)` | `POST /orgs/{id}/crawl` | `str`: the crawl id. |
| `get_crawl(crawl_id)` | `GET /orgs/{id}/crawls/{crawlId}` | `dict`: the crawl record. |
| `wait_for_crawl(crawl_id, *, timeout=600.0, poll_interval=2.0)` | polls `GET /orgs/{id}/crawls/{crawlId}` | `dict`: the finished crawl record. |
| `cancel_crawl(crawl_id)` | `DELETE /orgs/{id}/crawls/{crawlId}` | `None` |
| `crawl_errors(crawl_id)` | `GET /orgs/{id}/crawls/{crawlId}/errors` | `dict`: failed and robots-blocked pages. |
| `batch_scrape(urls, *, wait=True, timeout=600.0, **options)` | `POST /orgs/{id}/batch/scrape` | `dict` (finished record) or `str` (job id) with `wait=False`. |
| `map(url, *, search=None, limit=None, max_depth=None)` | `POST /orgs/{id}/map` | `list[str]`: URLs. |
| `search(query, **options)` | `POST /orgs/{id}/search` | `dict`: the search response. |
| `research(query, *, depth="quick", output=None, schema=None, **options)` | `POST /orgs/{id}/research` | `dict`: the research result. |
| `extract(urls, *, prompt=None, schema=None, **options)` | `POST /orgs/{id}/extract` | `dict`: the extract response. |
| `llms_txt(url, *, max_urls=None, full_text=False)` | `POST /orgs/{id}/llmstxt` | `dict`: the generated files. |

### whoami

```python
xdive.whoami()
```

```json
{
  "auth": "api_key",
  "keyId": "key_9b2e4f71c0a35d8e6f1a2b3c4d5e6f70",
  "orgId": "org_3f9a1c2e7b5d4a6f8e0c1b2a3d4e5f60",
  "scopes": ["crawl", "scrape", "search"],
  "plan": "free",
  "suspended": false
}
```

### scrape

```python
page = xdive.scrape(
    "https://example.com/pricing",
    formats=["markdown", "links"],
    max_age=3_600_000,  # accept a cached copy up to an hour old (milliseconds)
)
markdown = page_markdown(page)
print(sorted(page["artifacts"]))  # ['links', 'markdown']
```

Scrapes one page while you wait and returns the `data` object: `url`, `title`, `description`, `status_code`, the requested `artifacts` and more. Other options (`headers`, `respect_robots`, `redact_pii`, `actions`, …) are the [scrape parameters](https://xdive.dev/docs/scrape.md).

### crawl

```python
record = xdive.crawl("https://docs.example.com", max_pages=50, max_depth=2)
for page in record["results"]:
    print(page["url"], page.get("error") or len(page_markdown(page)))
```

Starts a crawl, then polls until it reaches `success`, `error` or `cancelled`, and returns the final record: `id`, `url`, `status`, `pages`, `durationMs`, `error`, `createdAt`, `results` and fetch statistics. A failed page stays in `results` with an `error` field. Without `formats`, each page carries its Markdown in `markdown`; with `formats`, in `artifacts`. Crawl options (`max_pages`, `max_depth`, `include_paths`, `exclude_paths`, `webhook_url`, …) are the [crawl parameters](https://xdive.dev/docs/crawl.md).

`timeout` (seconds) bounds the wait, not the crawl. When it runs out, `CrawlTimeout` is raised and the crawl keeps running on the server; call `cancel_crawl` to stop it.

To start a crawl and come back to it later, use the parts `crawl` is built from:

```python
crawl_id = xdive.start_crawl("https://docs.example.com", max_pages=500)
status = xdive.get_crawl(crawl_id)["status"]       # queued, running, success, error or cancelled
record = xdive.wait_for_crawl(crawl_id, timeout=1800)
```

### cancel_crawl

```python
xdive.cancel_crawl(crawl_id)
```

Cancels a queued or running crawl or batch scrape; the pages fetched so far are kept. A crawl that already finished raises `XDiveError` with status `409`.

### crawl_errors

```python
report = xdive.crawl_errors(crawl_id)
```

```json
{
  "id": "crl_5c8d2e1f4a7b4c3d9e0f1a2b3c4d5e6f",
  "status": "success",
  "error": "",
  "errors": [{ "url": "https://docs.example.com/old-page", "error": "HTTP 404" }],
  "robots_blocked": ["https://docs.example.com/private/"]
}
```

Lists the pages of a crawl or batch scrape that failed, without downloading the results. Pages skipped because of robots.txt are listed separately in `robots_blocked`.

### batch_scrape

```python
record = xdive.batch_scrape(["https://example.com/a", "https://example.com/b"], formats=["markdown"])

job_id = xdive.batch_scrape(urls, wait=False)  # returns immediately
record = xdive.wait_for_crawl(job_id)
```

Scrapes up to 100 URLs as one job. It runs through the crawl pipeline, so the result is a crawl record and the crawl methods (`get_crawl`, `cancel_crawl`, `crawl_errors`) work on its id. See [Batch scrape](https://xdive.dev/docs/batch-scrape.md).

### map

```python
urls = xdive.map("https://example.com", search="pricing", limit=100)
```

Returns the site's URLs, found from its sitemaps and links, without fetching their content. See [Map](https://xdive.dev/docs/map.md).

### search

```python
response = xdive.search("python packaging guide", limit=5, search_depth="advanced")
for result in response["results"]:
    print(result["rank"], result["url"])
    for passage in result.get("passages", []):
        print("  ", passage["text"][:120])
```

Returns the search response: `query`, `provider` and `results`. With `search_depth="advanced"`, the top results also carry `passages` (the excerpts most relevant to the query) and a `relevance_score`; results stay in the search engine's order. Other options (`sources`, `country`, `freshness`, `include_domains`, `auto_parameters`, `include_answer`, `scrape`, …) are the [search parameters](https://xdive.dev/docs/search.md).

### research

```python
result = xdive.research("What does PEP 639 change about license metadata?", depth="quick")
print(result["answer"])
for citation in result["citations"]:
    print(citation["number"], citation["url"])
```

Answers a question from several web sources with verified citations. `depth` is `"quick"` (seconds) or `"deep"` (one to two minutes, planned sub-queries). `output` is `"answer"` or `"report"`. Pass `schema` (a JSON Schema) to get structured `data` instead of prose, and `no_cache=True` to bypass the short-lived result cache (`cached` is `true` on a cache hit). The result has `answer` or `data`, `citations`, `sources`, `queries`, `confidence` and `usage`. In `usage`, `plan` (when present) says how the question was planned: `"single"` (searched directly), `"multi_hop"` (answered through an intermediate search) or `"unavailable"` (the planning step was too slow or failed). See [Research](https://xdive.dev/docs/research.md).

### extract

```python
result = xdive.extract(
    ["https://example.com/pricing", "https://example.com/plans"],
    prompt="List each plan with its monthly price",
)
```

Extracts structured data from up to 10 URLs with a `prompt`, a JSON Schema in `schema`, or both. See [Extract](https://xdive.dev/docs/extract.md).

### llms_txt

```python
files = xdive.llms_txt("https://docs.example.com", max_urls=20, full_text=True)
print(files["llms_txt"])
```

Generates an llms.txt index for a site, and with `full_text=True` also `llms_full_txt`. See [llms.txt](https://xdive.dev/docs/llms-txt.md).

## Async client

`AsyncXDive` has the same constructor and the same methods, as coroutines. Use it as an async context manager so its connection pool is closed:

```python
import asyncio

from xdive import AsyncXDive

async def main() -> None:
    async with AsyncXDive() as client:
        urls = ["https://example.com/a", "https://example.com/b", "https://example.com/c"]
        pages = await asyncio.gather(*(client.scrape(url) for url in urls))
        for page in pages:
            print(page["url"], page["title"])

asyncio.run(main())
```

One difference: on `AsyncXDive`, `org_id` is a coroutine method, `await client.org_id()`, not a property.

Concurrent requests count against your plan's concurrency limit; requests over it are answered with `429` and retried as described below.

## Errors

All exceptions derive from `XDiveError`, importable from `xdive`.

| Exception | Raised when | Attributes |
|---|---|---|
| `XDiveError` | The API answers with status `400` or above. | `status` (int), `message` (str), `code` (str or `None`) |
| `RateLimited` | Still overloaded (`429`, or `503` with code `SERVER_BUSY`) after `max_retries` retries. | `retry_after` (float, seconds), plus the above |
| `QuotaExceeded` | The API answers `402`: the organization's monthly quota is used up. | as `XDiveError` |
| `CrawlTimeout` | `crawl`, `wait_for_crawl` or `batch_scrape` waited longer than `timeout`. `status` is `408`; no HTTP request failed. | `crawl_id` |

`message` is the API's `error` field. `code` is the API's machine-readable `code` when it sends one, for example `CONCURRENCY_LIMIT` or `SERVER_BUSY`. `str(error)` reads `429 CONCURRENCY_LIMIT: too many concurrent requests…`, or `402: …` without a code.

```python
from xdive import XDive, XDiveError, QuotaExceeded, RateLimited

xdive = XDive()
try:
    page = xdive.scrape("https://example.com")
except RateLimited as error:
    print(f"still throttled, retry in {error.retry_after:.0f}s")
except QuotaExceeded:
    print("monthly quota used up")
except XDiveError as error:
    print(error.status, error.code, error.message)
```

Network failures and request timeouts are not wrapped: they raise `httpx` exceptions (`httpx.TimeoutException`, `httpx.ConnectError`, …). See [Errors and limits](https://xdive.dev/docs/errors.md) for the API's status codes.

## Retries and rate limits

The client retries only overload, the API's way of saying "slow down":

- `429`: the organization exceeded its plan's request rate or concurrency.
- `503` with the code `SERVER_BUSY` in the JSON body: the server is at capacity.

Both carry a `Retry-After` header. The client sleeps for that many seconds (1 second when the header is missing) and sends the request again, up to `max_retries` times, so a request is attempted at most `max_retries + 1` times. When the last attempt is still throttled it raises `RateLimited` with the final `Retry-After` in `retry_after`.

Every other error is raised at once, including any other `503`, such as a feature the server has not configured or a job queue that is unavailable: you get a plain `XDiveError` with the API's `status`, `message` and `code`. `max_retries=0` turns retries off. Retries are synchronous sleeps on `XDive` and `asyncio.sleep` on `AsyncXDive`.

## Crawl polling

`crawl`, `wait_for_crawl` and `batch_scrape` (with `wait=True`) poll `GET /orgs/{id}/crawls/{crawlId}`:

- Each poll sends the `ETag` of the last response in `If-None-Match`. While nothing changed, the server answers `304` without a body.
- Between polls the client waits the number of seconds in the server's `Retry-After` header, which is `2` while a crawl runs, and falls back to `poll_interval` when the header is absent. The wait never runs past the deadline.
- Polling stops at `success`, `error` or `cancelled` and returns that record. A crawl that ends in `error` is returned, not raised; check `record["status"]`.
- After `timeout` seconds it raises `CrawlTimeout`. The crawl itself is not cancelled.

## Closing the client

Each client holds an `httpx` connection pool. Close it when you are done, either with a context manager or explicitly:

```python
with XDive() as xdive:
    xdive.scrape("https://example.com")

xdive = XDive()
try:
    xdive.scrape("https://example.com")
finally:
    xdive.close()
```

For `AsyncXDive`, use `async with AsyncXDive() as client:` or `await client.aclose()`. A long-running service can create one client at startup, reuse it for every call, and close it at shutdown; reusing it also reuses the cached organization id and open connections.

## Self-hosted

Point the client at your own server with `base_url`, including the `/api` prefix, or set `XDIVE_BASE_URL`:

```python
xdive = XDive(api_key="ck_live_…", base_url="https://your-host/api")
```

```bash
export XDIVE_BASE_URL="https://your-host/api"
```

Keys are issued by the server they are used against. See [Self-hosting](https://xdive.dev/docs/self-hosting.md).

## Related

- [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md)
- [LangChain](https://xdive.dev/docs/langchain.md) and [LlamaIndex](https://xdive.dev/docs/llamaindex.md) integrations
- [Errors and limits](https://xdive.dev/docs/errors.md)

---

Source: https://xdive.dev/docs/sdk-python · All docs for agents: https://xdive.dev/llms.txt


---

# JavaScript SDK

The `xdive` package is a TypeScript client for Node.js and other JavaScript runtimes. Each method makes one API call, or polls until a job finishes, and resolves to the parsed JSON with TypeScript types. The client discovers your organization from the API key, retries rate-limited requests, and throws typed errors.

## Install

```bash
npm install xdive
pnpm add xdive
yarn add xdive
bun add xdive
```

Runtime requirements:

- **Node.js 18 or later**, or any runtime with a global `fetch`, `AbortController` and `setTimeout`. On a runtime without a global `fetch`, pass one in the [`fetch` option](#custom-fetch).
- **ES modules.** The package is ESM only. From CommonJS, load it with `await import("xdive")`.
- **TypeScript** declarations are included. To resolve the `xdive/langchain` entry point, use `"moduleResolution": "node16"`, `"nodenext"` or `"bundler"`.

The package has no runtime dependencies. The [LangChain.js tools](https://xdive.dev/docs/langchain.md) need `@langchain/core` and `zod`, declared as optional peer dependencies.

> **Warning:**
>
> An API key gives access to your organization's quota. Use the client on a server, not in code shipped to browsers.

## Configure the client

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY and, if set, XDIVE_BASE_URL

const configured = new XDive({
  apiKey: process.env.XDIVE_API_KEY,
  baseUrl: "https://xdive.dev/api",
  orgId: "org_3f9a1c2e7b5d4a6f8e0c1b2a3d4e5f60",
  timeoutMs: 60_000,
  maxRetries: 5,
});
```

`new XDive(options?: XDiveOptions)` accepts:

| Option | Type | Environment variable | Default | Description |
|---|---|---|---|---|
| `apiKey` | `string` | `XDIVE_API_KEY` | required | Organization API key (`ck_live_…`). The constructor throws when neither is set. |
| `baseUrl` | `string` | `XDIVE_BASE_URL` | `https://xdive.dev/api` | API base URL, including `/api`. Trailing slashes are removed. |
| `orgId` | `string` | none | discovered | Organization id. Pass it to skip the `/whoami` lookup. |
| `timeoutMs` | `number` | none | `120000` | Per-request timeout in milliseconds. |
| `maxRetries` | `number` | none | `3` | How many times an overload response (`429`, or `503` with code `SERVER_BUSY`) is retried. See [Retries](#retries-and-rate-limits). |
| `fetch` | `typeof fetch` | none | `globalThis.fetch` | The `fetch` implementation to use. See [Custom fetch](#custom-fetch). |

Environment variables are read from `process.env` when the runtime has one. Every request sends the key in the `X-API-Key` header.

An API key belongs to one organization, and organization endpoints live under `/orgs/{id}/`. The first call that needs the organization id calls `GET /whoami` and caches the `orgId` for the life of the client; `await xdive.orgId()` returns it.

The key's scopes decide which methods work: `scrape` for scraping, `crawl` for crawls and maps, `search` for search and research. A call outside the key's scopes rejects with a `XDiveError` with status `403`. See [Authentication](https://xdive.dev/docs/authentication.md).

## A first request

```ts
import { XDive, pageMarkdown } from "xdive";

const xdive = new XDive();
const page = await xdive.scrape("https://example.com");
console.log(page.title);
console.log(pageMarkdown(page));
```

`scrape` resolves to the page object from the response's `data` field. Each requested format is an entry under `artifacts`; Markdown is the default format. `pageMarkdown(page)` returns the Markdown whether it is in `artifacts.markdown.content` (scrape, or a crawl with `formats`) or in `markdown` (a crawl without `formats`), and an empty string when there is none.

## Methods

Every method returns a promise. Options are sent as the API's JSON fields, unchanged, so every field in the [API reference](https://xdive.dev/docs/scrape.md) is available even when the type does not name it. Options that are `undefined` or `null` are left out of the request.

`formats` takes the API's format names (the `FormatName` type): `markdown`, `text`, `html`, `rawHtml`, `links`, `images`, `json`, `summary`, `question`, `highlights`, `branding`, `audio`, `screenshot`, `product` and `menu` (`audio` and `screenshot` on `scrape` only). A name is sent as `{ type: "markdown" }`; pass an object, such as `{ type: "json", schema: { … } }` or `{ type: "question", question: "…" }`, to set a format's options.

| Method | API call | Resolves to |
|---|---|---|
| `orgId()` | `GET /whoami`, once | `string` |
| `whoami()` | `GET /whoami` | `Json` |
| `scrape(url, options?)` | `POST /orgs/{id}/scrape` | `Page` |
| `crawl(url, options?)` | `POST /orgs/{id}/crawl`, then polls | `CrawlRecord` (finished) |
| `startCrawl(url, options?)` | `POST /orgs/{id}/crawl` | `string` (crawl id) |
| `getCrawl(crawlId)` | `GET /orgs/{id}/crawls/{crawlId}` | `CrawlRecord` |
| `waitForCrawl(crawlId, options?)` | polls `GET /orgs/{id}/crawls/{crawlId}` | `CrawlRecord` (finished) |
| `cancelCrawl(crawlId)` | `DELETE /orgs/{id}/crawls/{crawlId}` | `void` |
| `crawlErrors(crawlId)` | `GET /orgs/{id}/crawls/{crawlId}/errors` | `CrawlErrors` |
| `batchScrape(urls, options?)` | `POST /orgs/{id}/batch/scrape`, then polls | `CrawlRecord` (finished), or `string` (job id) with `wait: false` |
| `map(url, options?)` | `POST /orgs/{id}/map` | `string[]` |
| `search(query, options?)` | `POST /orgs/{id}/search` | `SearchResponse` |
| `research(query, options?)` | `POST /orgs/{id}/research` | `ResearchResult` |
| `extract(urls, options?)` | `POST /orgs/{id}/extract` | `Json` |
| `llmsTxt(url, options?)` | `POST /orgs/{id}/llmstxt` | `Json` |

### whoami and orgId

```ts
const me = await xdive.whoami();
// { auth: "api_key", keyId: "key_9b2e…", orgId: "org_3f9a…", scopes: ["crawl", "scrape", "search"], plan: "free", suspended: false }
const orgId = await xdive.orgId();
```

### scrape

```ts
const page = await xdive.scrape("https://example.com/pricing", {
  formats: ["markdown", "links"],
  max_age: 3_600_000, // accept a cached copy up to an hour old (milliseconds)
});
```

`scrape(url: string, options?: { formats?: Format[] } & Json): Promise<Page>`. Other options (`headers`, `respect_robots`, `redact_pii`, `actions`, …) are the [scrape parameters](https://xdive.dev/docs/scrape.md).

### crawl

```ts
const record = await xdive.crawl("https://docs.example.com", { max_pages: 50, max_depth: 2 });
for (const page of record.results ?? []) {
  console.log(page.url, page.error ?? pageMarkdown(page).length);
}
```

`crawl(url: string, options?: { formats?: Format[]; timeoutMs?: number; pollIntervalMs?: number } & Json): Promise<CrawlRecord>` starts a crawl, then polls until it reaches `success`, `error` or `cancelled`. `timeoutMs` (default `600000`) bounds the wait, not the crawl: when it runs out the promise rejects with `CrawlTimeoutError` and the crawl keeps running on the server. `pollIntervalMs` (default `2000`) applies only when the server sends no `Retry-After`. A failed page stays in `results` with an `error` field. Crawl options (`max_pages`, `max_depth`, `include_paths`, `exclude_paths`, `webhook_url`, …) are the [crawl parameters](https://xdive.dev/docs/crawl.md).

To start a crawl and come back to it later:

```ts
const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 500 });
const { status } = await xdive.getCrawl(crawlId); // "queued" | "running" | "success" | "error" | "cancelled"
const finished = await xdive.waitForCrawl(crawlId, { timeoutMs: 1_800_000, pollIntervalMs: 5000 });
```

### cancelCrawl

```ts
await xdive.cancelCrawl(crawlId);
```

Cancels a queued or running crawl or batch scrape; the pages fetched so far are kept. A crawl that already finished rejects with a `XDiveError` with status `409`.

### crawlErrors

```ts
const report = await xdive.crawlErrors(crawlId);
for (const failure of report.errors) console.log(failure.url, failure.error);
console.log("blocked by robots.txt:", report.robots_blocked);
```

Lists the pages of a crawl or batch scrape that failed, without downloading the results.

### batchScrape

```ts
const record = await xdive.batchScrape(["https://example.com/a", "https://example.com/b"], {
  formats: ["markdown"],
  timeoutMs: 300_000,
});
```

`batchScrape(urls: string[], options?: { formats?: Format[]; timeoutMs?: number; wait?: boolean } & Json)` scrapes up to 100 URLs as one job. By default it waits for the job and resolves to the finished `CrawlRecord`. With `wait: false` it resolves to the job id as soon as the job is accepted; the overloads type that result as `string`:

```ts
const jobId = await xdive.batchScrape(urls, { wait: false }); // string
const record = await xdive.waitForCrawl(jobId);
```

The job is a crawl, so `getCrawl`, `waitForCrawl`, `cancelCrawl` and `crawlErrors` work on its id. See [Batch scrape](https://xdive.dev/docs/batch-scrape.md).

### map

```ts
const urls = await xdive.map("https://example.com", { search: "pricing", limit: 100 });
```

`map(url: string, options?: { search?: string; limit?: number; max_depth?: number }): Promise<string[]>` returns the site's URLs, found from its sitemaps and links, without fetching their content. See [Map](https://xdive.dev/docs/map.md).

### search

```ts
const response = await xdive.search("node.js test runner", { limit: 5, search_depth: "advanced" });
for (const result of response.results) {
  console.log(result.rank, result.url, result.relevance_score);
  for (const passage of result.passages ?? []) console.log("  ", passage.text.slice(0, 120));
}
```

`search(query: string, options?: Json): Promise<SearchResponse>`. With `search_depth: "advanced"`, the top results also carry `passages` (the excerpts most relevant to the query) and a `relevance_score`; results stay in the search engine's order. Other options (`sources`, `country`, `freshness`, `include_domains`, `auto_parameters`, `include_answer`, `scrape`, …) are the [search parameters](https://xdive.dev/docs/search.md).

### research

```ts
const result = await xdive.research("What does PEP 639 change about license metadata?", { depth: "quick" });
console.log(result.answer);
for (const citation of result.citations) console.log(citation.number, citation.url);
```

`research(query: string, options?: { depth?: "quick" | "deep"; output?: "answer" | "report"; schema?: Json; no_cache?: boolean } & Json): Promise<ResearchResult>`. `depth` defaults to `"quick"` (seconds); `"deep"` takes one to two minutes and plans sub-queries. Pass `schema` (a JSON Schema) to get structured `data` instead of prose, and `no_cache: true` to bypass the short-lived result cache (`cached` is `true` on a cache hit). `usage.plan`, when present, says how the question was planned: `"single"` (searched directly), `"multi_hop"` (answered through an intermediate search) or `"unavailable"` (the planning step was too slow or failed). See [Research](https://xdive.dev/docs/research.md).

### extract

```ts
const result = await xdive.extract(["https://example.com/pricing", "https://example.com/plans"], {
  prompt: "List each plan with its monthly price",
});
```

`extract(urls: string[], options?: { prompt?: string; schema?: Json } & Json): Promise<Json>` extracts structured data from up to 10 URLs. See [Extract](https://xdive.dev/docs/extract.md).

### llmsTxt

```ts
const files = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true });
console.log(files.llms_txt);
```

`llmsTxt(url: string, options?: { max_urls?: number; full_text?: boolean }): Promise<Json>` generates an llms.txt index, and with `full_text` also `llms_full_txt`. See [llms.txt](https://xdive.dev/docs/llms-txt.md).

## Types

The package exports these types. Response types list the fields the SDK relies on and allow any other field the API returns.

| Type | Describes |
|---|---|
| `XDiveOptions` | The constructor options above. |
| `Json` | `Record<string, unknown>`, used for options and untyped responses. |
| `FormatName` | The format names listed under [Methods](#methods); `"raw_html"` is accepted as an alias of `"rawHtml"`. |
| `Format` | A `FormatName`, or an object `{ type: FormatName, prompt?, schema?, question?, query?, max_results? }` that also sets the format's options. |
| `Page` | A scraped page: `url`, `title`, `description`, `markdown`, `artifacts`, `error`. |
| `CrawlRecord` | A crawl or batch scrape: `id`, `status`, `pages`, `error`, `results` (`Page[]`). |
| `CrawlErrors` | `id`, `status`, `error`, `errors` (`url` and `error` pairs), `robots_blocked` (URLs). |
| `SearchResponse` | `query`, `provider`, `results`, and `answer` and `parameters` when requested. |
| `SearchResult` | `id`, `source`, `rank`, `title`, `url`, `description`, `published_at`, `page`, `passages`, `relevance_score`. |
| `Passage` | `text`, `score`, `start` (offset in the source text). |
| `ResearchResult` | `query`, `answer` or `data`, `citations`, `sources`, `queries`, `confidence`, `usage` (with `plan`), `cached`. |
| `Citation` | `number`, `url`, `title`, `quote`. |

```ts
import type { CrawlRecord, Page, ResearchResult } from "xdive";
```

The constant `DEFAULT_BASE_URL` holds the hosted API base URL.

## Errors

All error classes extend `XDiveError`, which extends `Error`.

| Class | Thrown when | Properties |
|---|---|---|
| `XDiveError` | The API answers with status `400` or above. | `status` (number), `code` (string or `undefined`), `message` |
| `RateLimitedError` | Still overloaded (`429`, or `503` with code `SERVER_BUSY`) after `maxRetries` retries. | `retryAfterSeconds` (number), plus the above |
| `QuotaExceededError` | The API answers `402`: the organization's monthly quota is used up. | as `XDiveError` |
| `CrawlTimeoutError` | `crawl`, `waitForCrawl` or `batchScrape` waited longer than `timeoutMs`. `status` is `408`; no HTTP request failed. | `crawlId` |

`code` is the API's machine-readable code when it sends one, for example `CONCURRENCY_LIMIT` or `SERVER_BUSY`. `message` combines the status, the code and the API's `error` text: `429 CONCURRENCY_LIMIT: too many concurrent requests…`, or `402: …` without a code.

```ts
import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive";

const xdive = new XDive();
try {
  const page = await xdive.scrape("https://example.com");
} catch (error) {
  if (error instanceof RateLimitedError) console.log(`still throttled, retry in ${error.retryAfterSeconds}s`);
  else if (error instanceof QuotaExceededError) console.log("monthly quota used up");
  else if (error instanceof XDiveError) console.log(error.status, error.code, error.message);
  else throw error; // network failure or timeout
}
```

Network failures and timeouts are not wrapped: the promise rejects with whatever `fetch` threw. A request that exceeds `timeoutMs` is aborted, which rejects with an `AbortError`. See [Errors and limits](https://xdive.dev/docs/errors.md) for the API's status codes.

## Retries and rate limits

The client retries only overload, the API's way of saying "slow down":

- `429`: the organization exceeded its plan's request rate or concurrency.
- `503` with the code `SERVER_BUSY` in the JSON body: the server is at capacity.

Both carry a `Retry-After` header. The client waits that many seconds (1 second when the header is missing) and sends the request again, up to `maxRetries` times, so a request is attempted at most `maxRetries + 1` times; each attempt gets its own `timeoutMs`. When the last attempt is still throttled the promise rejects with `RateLimitedError`, whose `retryAfterSeconds` is the final `Retry-After`.

Every other error rejects at once, including any other `503`, such as a feature the server has not configured or a job queue that is unavailable: you get a plain `XDiveError` with the API's `status`, `message` and `code`. `maxRetries: 0` turns retries off. `maxRetries` is a public property, so you can change it on an existing client.

## Crawl polling

`crawl`, `waitForCrawl` and `batchScrape` poll `GET /orgs/{id}/crawls/{crawlId}`:

- Each poll sends the `ETag` of the last response in `If-None-Match`. While nothing changed, the server answers `304` without a body.
- Between polls the client waits the number of seconds in the server's `Retry-After` header, which is `2` while a crawl runs, and falls back to `pollIntervalMs` when the header is absent. The wait never runs past the deadline.
- Polling stops at `success`, `error` or `cancelled` and resolves to that record. A crawl that ends in `error` resolves; check `record.status`.
- After `timeoutMs` it rejects with `CrawlTimeoutError`. The crawl itself is not cancelled.

## Custom fetch

Pass `fetch` to log or instrument requests, to route them through a proxy, or to provide `fetch` on a runtime that lacks a global one. The client calls it with a URL string and a `RequestInit`, and expects a standard `Response`.

```ts
import { XDive } from "xdive";

const xdive = new XDive({
  fetch: async (input, init) => {
    const started = Date.now();
    const response = await fetch(input, init);
    console.log(init?.method, String(input), response.status, `${Date.now() - started}ms`);
    return response;
  },
});
```

Tests can pass a function that returns canned `Response` objects instead of calling the network.

## Self-hosted

Point the client at your own server with `baseUrl`, including the `/api` prefix, or set `XDIVE_BASE_URL`:

```ts
const xdive = new XDive({ apiKey: process.env.XDIVE_API_KEY, baseUrl: "https://your-host/api" });
```

Keys are issued by the server they are used against. See [Self-hosting](https://xdive.dev/docs/self-hosting.md).

## Related

- [Python SDK](https://xdive.dev/docs/sdk-python.md)
- [LangChain](https://xdive.dev/docs/langchain.md)
- [Errors and limits](https://xdive.dev/docs/errors.md)

---

Source: https://xdive.dev/docs/sdk-javascript · All docs for agents: https://xdive.dev/llms.txt


---

# Go SDK

`github.com/xdive/xdive-go` is the official Go client. It uses only the standard library, needs Go 1.22 or newer, takes a `context.Context` on every call, and returns typed results and errors. Each method makes one API call, or polls until a job finishes. The client discovers your organization from the API key, retries rate-limited requests, and sends an `Idempotency-Key` with every POST so a retry never runs or bills the work twice.

## Install

```bash
go get github.com/xdive/xdive-go
```

## Configuration

| Setting  | Option           | Environment variable                      |
| -------- | ---------------- | ----------------------------------------- |
| API key  | `WithAPIKey`     | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`)    |
| Base URL | `WithBaseURL`    | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`)  |

Options win over the environment. The default base URL is
`xdive.DefaultBaseURL`. One API key identifies one organization; the client
looks the organization id up once with `GET /whoami` and caches it (skip the
lookup with `WithOrgID`). Other options: `WithHTTPClient`, `WithMaxRetries`
(default 3) and `WithTimeout` (per attempt, default 120s).

## Quickstart

```go
package main

import (
	"context"
	"fmt"
	"log"

	xdive "github.com/xdive/xdive-go"
)

func main() {
	ctx := context.Background()
	client, err := xdive.New() // reads XDIVE_API_KEY
	if err != nil {
		log.Fatal(err)
	}

	// Scrape one page.
	page, err := client.Scrape(ctx, "https://example.com", &xdive.ScrapeOptions{
		Formats: xdive.F(xdive.FormatMarkdown),
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(xdive.PageMarkdown(page))

	// Crawl a site and wait for it to finish.
	record, err := client.Crawl(ctx, "https://example.com", &xdive.CrawlOptions{
		Formats: xdive.F(xdive.FormatMarkdown),
		Extra:   xdive.Map{"limit": 20},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(record.Status, len(record.Results))

	// Search the web.
	res, err := client.Search(ctx, "go context cancellation", xdive.Map{"limit": 5})
	if err != nil {
		log.Fatal(err)
	}
	for _, r := range res.Results {
		fmt.Println(r.Title, r.URL)
	}

	// Research a question with verified citations.
	answer, err := client.Research(ctx, "What changed in Go 1.22?", nil)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(answer.Answer, answer.Confidence)
	for _, c := range answer.Citations {
		fmt.Printf("[%d] %s\n", c.Number, c.URL)
	}
}
```

Every method takes a `context.Context` first and a pointer to an options struct
(`nil` for defaults). Options structs have an `Extra xdive.Map` for API fields
the SDK does not model, so new API fields never need an SDK upgrade.

### Methods

| Method | Returns |
| ------ | ------- |
| `Scrape(ctx, url, *ScrapeOptions)` | `*Page` |
| `StartCrawl(ctx, url, *CrawlOptions)` | crawl id |
| `Crawl(ctx, url, *CrawlOptions)` | `*CrawlRecord` (waits) |
| `GetCrawl(ctx, id)` | `*CrawlRecord` |
| `WaitForCrawl(ctx, id, *WaitOptions)` | `*CrawlRecord` (ETag and `Retry-After` aware) |
| `CancelCrawl(ctx, id)` | `error` |
| `CrawlErrors(ctx, id)` | `*CrawlErrors` (failed and robots.txt-blocked pages) |
| `BatchScrape(ctx, urls, *BatchOptions)` | `*CrawlRecord` (waits) |
| `StartBatchScrape(ctx, urls, *BatchOptions)` | job id, no waiting |
| `Map(ctx, url, *MapOptions)` | `[]string` |
| `Search(ctx, query, xdive.Map)` | `*SearchResponse` |
| `Research(ctx, query, *ResearchOptions)` | `*ResearchResult` |
| `Extract(ctx, urls, *ExtractOptions)` | `xdive.Map` |
| `LlmsTxt(ctx, url, *LlmsTxtOptions)` | `xdive.Map` |
| `Whoami(ctx)`, `OrgID(ctx)` | key details, organization id |

Every POST method also accepts trailing `CallOption`s.

## Error handling

Failed requests return `*xdive.Error`:

```go
_, err := client.Scrape(ctx, "https://example.com", nil)

var apiErr *xdive.Error
switch {
case err == nil:
case errors.Is(err, xdive.ErrQuotaExceeded): // HTTP 402
	log.Println("out of credits")
case errors.Is(err, xdive.ErrRateLimited): // retries exhausted
	var rl *xdive.RateLimitedError
	if errors.As(err, &rl) {
		log.Println("try again in", rl.RetryAfter)
	}
case errors.As(err, &apiErr):
	log.Println(apiErr.Status, apiErr.Code, apiErr.Message,
		apiErr.Retryable, apiErr.RequestID, apiErr.DocsURL)
}
```

`WaitForCrawl` (and `Crawl`, `BatchScrape`) return an error matching
`xdive.ErrCrawlTimeout` when the job does not finish in time; the job keeps
running, so call `CancelCrawl` if you no longer want it. Cancellation and
deadlines of your `ctx` are returned as `ctx.Err()`, also while the client is
waiting between retries.

Quote `RequestID` when contacting support.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code
  `IDEMPOTENCY_KEY_IN_PROGRESS` are retried after `Retry-After` seconds
  (default 1) up to `MaxRetries` times, then returned as `*RateLimitedError`.
- Transport errors and timeouts are retried for GET and POST with exponential
  backoff (1s, 2s, 4s, capped at 8s).
- No other status is retried.
- Every POST carries an `Idempotency-Key` (random UUID v4), generated once per
  call and reused by each retry, so a retry never runs or bills the work
  twice. Pass your own to make a whole operation safe to repeat:

```go
id, err := client.StartCrawl(ctx, "https://example.com", nil,
	xdive.WithIdempotencyKey("nightly-crawl-2026-01-01"))
```

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python SDK](https://xdive.dev/docs/sdk-python.md) and [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md), which have the same methods.

---

Source: https://xdive.dev/docs/sdk-go · All docs for agents: https://xdive.dev/llms.txt


---

# Java SDK

Official Java client for the [XDive](https://xdive.dev) web-data
API: scrape, crawl, map, search, research and extract. Requires Java 11 or newer,
works from Kotlin and Scala, and depends only on Jackson databind
(HTTP is `java.net.http.HttpClient`).

## Install

Maven:

```xml
<dependency>
  <groupId>dev.xdive</groupId>
  <artifactId>xdive</artifactId>
  <version>1.5.0</version>
</dependency>
```

Gradle (Kotlin DSL):

```kotlin
dependencies {
    implementation("dev.xdive:xdive:1.5.0")
}
```

## Quickstart

```java
import dev.xdive.*;
import java.util.List;

public class Quickstart {
    public static void main(String[] args) {
        // Reads XDIVE_API_KEY (and XDIVE_BASE_URL) from the environment.
        XDive xdive = XDive.builder().build();

        // Scrape one page.
        Page page = xdive.scrape("https://example.com",
                ScrapeOptions.builder().formats("markdown", "links").build());
        System.out.println(XDive.pageMarkdown(page));

        // Crawl a site and wait for the result.
        CrawlRecord crawl = xdive.crawl("https://example.com",
                CrawlOptions.builder().formats("markdown").extra("limit", 20).build());
        for (Page p : crawl.getResults()) {
            System.out.println(p.getUrl() + " " + XDive.pageMarkdown(p).length());
        }

        // Search the web.
        SearchResponse results = xdive.search("open source web crawlers",
                SearchOptions.builder().limit(5).build());
        results.getResults().forEach(r -> System.out.println(r.getTitle() + " " + r.getUrl()));

        // Research a question with citations.
        ResearchResult answer = xdive.research("What is robots.txt?");
        System.out.println(answer.getAnswer());
        answer.getCitations().forEach(c -> System.out.println("[" + c.getNumber() + "] " + c.getUrl()));
    }
}
```

Other calls: `startCrawl`, `getCrawl`, `waitForCrawl`, `cancelCrawl`, `crawlErrors`,
`startBatchScrape`, `batchScrape`, `map`, `extract`, `llmsTxt`, `whoami`.

Every main call has a `CompletableFuture` variant (`scrapeAsync`, `searchAsync`,
`researchAsync`, `crawlAsync`, ...). A failed future completes with a
`CompletionException` whose cause is the `XDiveException`.

### Formats

Plain formats are strings (`formats("markdown", "links")`). Formats with options
are objects:

```java
Format json = Format.builder("json")
        .prompt("Extract the product name and price")
        .schema(Map.of("type", "object"))
        .build();
Page p = xdive.scrape(url, ScrapeOptions.builder().formats(Format.of("markdown"), json).build());
```

`XDive.pageMarkdown(page)` returns the Markdown whichever field carries it.

### Fields the SDK does not model yet

Every options builder has `extra(Map)` / `extra(key, value)`, merged into the
request body (and winning over modeled fields). Every result keeps the complete
response in `getRaw()` (a Jackson `JsonNode`), and unknown fields never break
parsing.

## Configuration

| Setting | Builder | Environment | Default |
| --- | --- | --- | --- |
| API key | `apiKey(..)` | `XDIVE_API_KEY`, then `CRAWLER_API_KEY` | required |
| API origin | `baseUrl(..)` | `XDIVE_BASE_URL`, then `CRAWLER_BASE_URL` | `XDive.DEFAULT_BASE_URL` |
| Organization id | `orgId(..)` | none | discovered once via `GET /whoami`, cached |
| Retries | `maxRetries(n)` | none | 3 |
| Per-attempt timeout | `timeout(Duration)` | none | 120 s |
| HTTP client | `httpClient(..)` | none | `HttpClient.newHttpClient()` |

The client is thread-safe: create one and share it.

## Errors

Failures are unchecked `XDiveException`s:

```java
try {
    xdive.scrape("https://example.com");
} catch (RateLimitedException e) {
    // The API kept saying "slow down" until the retries ran out.
    System.out.println("try again in " + e.getRetryAfter());
} catch (QuotaExceededException e) {
    // HTTP 402: the plan quota is used up.
} catch (XDiveException e) {
    System.out.println(e.getStatus() + " " + e.getCode() + ": " + e.getMessage());
    System.out.println("retryable=" + e.isRetryable()
            + " requestId=" + e.getRequestId() + " docs=" + e.getDocsUrl());
}
```

`CrawlTimeoutException` (HTTP status 408, `getCrawlId()`) is thrown when
`waitForCrawl` / `crawl` / `batchScrape` times out; the crawl keeps running on the
server. Transport failures that outlast the retries surface as an
`XDiveException` with status `0` and code `NETWORK_ERROR`. A missing API key makes
`build()` throw `IllegalStateException`.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code
  `IDEMPOTENCY_KEY_IN_PROGRESS` are retried after the `Retry-After` seconds
  (default 1, fractions allowed), up to `maxRetries`, then `RateLimitedException`.
- Transport errors and timeouts are retried for GET and POST with a 1 s, 2 s, 4 s
  ... backoff capped at 8 s.
- No other status is retried.
- Every POST carries an `Idempotency-Key` header (a random UUID generated once
  per call and reused by each retry), so a retry never runs or bills the work
  twice. A new call gets a new key. To set your own:
  `ScrapeOptions.builder().idempotencyKey("order-1234-scrape").build()`.

## Kotlin

```kotlin
import dev.xdive.CrawlOptions
import dev.xdive.XDive

fun main() {
    val xdive = XDive.builder().apiKey(System.getenv("XDIVE_API_KEY")).build()

    val page = xdive.scrape("https://example.com")
    println(XDive.pageMarkdown(page))

    val crawl = xdive.crawl(
        "https://example.com",
        CrawlOptions.builder().formats("markdown").extra("limit", 10).build(),
    )
    crawl.results.forEach { println(it.url) }
}
```

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-java · All docs for agents: https://xdive.dev/llms.txt


---

# .NET SDK

The official .NET client for the XDive web data API: scrape, crawl, map, search, research and
extract web pages as clean Markdown and structured data.

- Targets `netstandard2.0` and `net8.0`: .NET Framework 4.6.2+, .NET 6/7/8/9, Unity, Xamarin.
- Async everywhere, `CancellationToken` last, nullable reference types, XML docs.
- Safe retries: `Retry-After` is honored and every POST carries an `Idempotency-Key`.
- Works with `IHttpClientFactory` and dependency injection.

## Install

```sh
dotnet add package XDive
```

## Quickstart

```csharp
using XDive;

using var client = XDiveClient.FromEnvironment(); // reads XDIVE_API_KEY

// Scrape one page.
Page page = await client.ScrapeAsync("https://example.com",
    new ScrapeOptions { Formats = Format.Of("markdown") });
Console.WriteLine(XDiveClient.PageMarkdown(page));

// Crawl a site and wait for it to finish.
CrawlRecord crawl = await client.CrawlAsync("https://example.com", new CrawlOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["limit"] = 20 },
});
Console.WriteLine($"{crawl.Status}: {crawl.Results.Count} pages");

// Search the web.
SearchResponse found = await client.SearchAsync("dotnet cancellation tokens",
    new SearchOptions { Limit = 5 });
foreach (var r in found.Results) Console.WriteLine($"{r.Title} {r.Url}");

// Research a question with verified citations.
ResearchResult answer = await client.ResearchAsync("What changed in .NET 8?");
Console.WriteLine(answer.Answer);
foreach (var c in answer.Citations) Console.WriteLine($"[{c.Number}] {c.Url}");
```

Other calls: `StartCrawlAsync` / `GetCrawlAsync` / `WaitForCrawlAsync` / `CancelCrawlAsync` /
`CrawlErrorsAsync`, `StartBatchScrapeAsync` / `BatchScrapeAsync` (up to 100 URLs), `MapAsync`,
`ExtractAsync`, `LlmsTxtAsync`, `WhoamiAsync`.

### Formats

`Format.Of("markdown", "links")` makes plain formats. A string converts implicitly, and object
formats carry their options:

```csharp
Formats = new List<Format>
{
    "markdown",
    Format.Json(new Dictionary<string, object?> { ["type"] = "object" }, prompt: "the product price"),
    new Format("highlights") { Query = "pricing", MaxResults = 5 },
}
```

### Fields the SDK does not model

Every options class has an `Extra` dictionary that is merged into the request body (and wins over
the typed fields). Every result keeps the complete response as `Raw` (a `JsonElement`), and unknown
JSON fields never break parsing:

```csharp
var page = await client.ScrapeAsync(url, new ScrapeOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["only_main_content"] = true },
});
var surprise = page.Raw.GetProperty("someNewField");
```

## Configuration

| Setting  | Option                | Environment variable                     |
| -------- | --------------------- | ---------------------------------------- |
| API key  | `XDiveOptions.ApiKey`  | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`)   |
| Base URL | `XDiveOptions.BaseUrl` | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`) |

Options win over the environment. The default base URL is `XDiveClient.DefaultBaseUrl`.

```csharp
var client = new XDiveClient(new XDiveOptions
{
    ApiKey = "ck_live_...",
    BaseUrl = "https://your-host/api",   // optional
    OrgId = "org_123",                    // optional: skips the GET /whoami lookup
    MaxRetries = 3,                       // default 3, 0 disables retries
    Timeout = TimeSpan.FromSeconds(120),  // per attempt, default 120s
    HttpClient = myHttpClient,            // optional, see below
});
```

One API key identifies one organization. The client looks the organization id up once with
`GET /whoami` and caches it (thread-safe); set `OrgId` to skip the lookup.

## Dependency injection

`XDiveClient` is thread-safe: register one instance. The optional package
`XDive.Extensions.DependencyInjection` does it on top of `IHttpClientFactory`:

```sh
dotnet add package XDive.Extensions.DependencyInjection
```

```csharp
builder.Services.AddXDive(o => o.ApiKey = builder.Configuration["XDive:ApiKey"]);

public class MyService(XDiveClient xdive)
{
    public Task<Page> Fetch(string url, CancellationToken ct) => xdive.ScrapeAsync(url, null, ct);
}
```

`AddXDive` returns the `IHttpClientBuilder`, so you can add handlers or resilience policies.

Without the extension package, register it yourself:

```csharp
builder.Services.AddHttpClient("xdive");
builder.Services.AddSingleton(sp => new XDiveClient(new XDiveOptions
{
    ApiKey = builder.Configuration["XDive:ApiKey"],
    HttpClient = sp.GetRequiredService<IHttpClientFactory>().CreateClient("xdive"),
}));
```

When you pass your own `HttpClient`, the client never disposes it and the `HttpClient.Timeout`
still applies (default 100s): raise it above `XDiveOptions.Timeout` for long calls such as
`ResearchAsync(..., new ResearchOptions { Depth = "deep" })`. `XDiveClient.Dispose()` only disposes
an `HttpClient` the client created itself.

## Errors

API errors are thrown as `XDiveException`:

```csharp
try
{
    await client.ScrapeAsync("https://example.com");
}
catch (QuotaExceededException e)          // HTTP 402: plan quota exhausted
{
    Console.WriteLine($"Upgrade your plan ({e.RequestId})");
}
catch (RateLimitedException e)            // retries ran out on 429 / SERVER_BUSY
{
    await Task.Delay(e.RetryAfter);
}
catch (XDiveException e)
{
    Console.WriteLine($"{e.Status} {e.Code}: {e.Message}");
    Console.WriteLine($"retryable={e.Retryable} request={e.RequestId} docs={e.DocsUrl}");
}
```

| Property    | Meaning                                                          |
| ----------- | ---------------------------------------------------------------- |
| `Status`    | HTTP status                                                      |
| `Code`      | machine-readable code, `null` for older error bodies             |
| `Retryable` | the server's `retryable` flag                                    |
| `RequestId` | the `X-Request-ID` header, quote it to support                   |
| `DocsUrl`   | documentation link for the error, when provided                  |

`CrawlTimeoutException` (with `CrawlId`) is thrown by `WaitForCrawlAsync` / `CrawlAsync` when the
crawl does not finish within `Timeout` (default 10 minutes); the crawl keeps running on the server.
A request that times out on every attempt throws `TimeoutException`; cancelling the token throws
`OperationCanceledException`.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS` are retried
  after `Retry-After` seconds (default 1, fractions allowed), up to `MaxRetries` times, then
  `RateLimitedException` is thrown.
- Transport errors and timeouts are retried for GET and POST with backoff 1s, 2s, 4s, capped at 8s.
- Other statuses are never retried.
- Every POST carries an `Idempotency-Key` header (a random GUID) generated once per call and reused
  by every retry, so a retry after a timeout never runs the work twice. Set your own with
  `new ScrapeOptions { IdempotencyKey = "order-1234" }`; calls with the same key return the first
  call's response.
- Every wait honors the `CancellationToken`.

## Waiting for crawls

`WaitForCrawlAsync` sends `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and it
follows the server's `Retry-After`:

```csharp
string id = await client.StartCrawlAsync("https://example.com");
CrawlRecord done = await client.WaitForCrawlAsync(id,
    new WaitOptions { Timeout = TimeSpan.FromMinutes(5) }, cancellationToken);
CrawlErrors errors = await client.CrawlErrorsAsync(id);
```

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-dotnet · All docs for agents: https://xdive.dev/llms.txt


---

# PHP SDK

The official PHP SDK for the [XDive](https://xdive.dev) web-data API: scrape, crawl, search and research the web as clean Markdown. PHP 8.1+, works in plain PHP, Laravel and Symfony. HTTP goes through PSR-18, so you can plug in any client; Guzzle is the default.

## Install

```bash
composer require xdive/xdive
```

## Quickstart

```php
<?php

use XDive\Client;
use XDive\Format;

require 'vendor/autoload.php';

$client = new Client(apiKey: getenv('XDIVE_API_KEY'));   // or Client::fromEnvironment()

// Scrape one page
$page = $client->scrape('https://example.com', [Format::MARKDOWN]);
echo $page->title, "\n", $page->markdown(), "\n";

// Crawl a site and wait for the result
$crawl = $client->crawl('https://example.com', [Format::MARKDOWN], ['limit' => 20, 'max_depth' => 2]);
foreach ($crawl->results as $p) {
    echo $p->url, ' ', strlen($p->markdown()), " chars\n";
}

// Search the web
$search = $client->search('php 8.4 release notes', limit: 5, includeAnswer: true);
foreach ($search->results as $hit) {
    echo "{$hit->rank}. {$hit->title} {$hit->url}\n";
}

// Research a question with verified citations
$research = $client->research('How does HTTP/3 differ from HTTP/2?', depth: 'quick');
echo $research->answer, "\n";
foreach ($research->citations as $c) {
    echo "[{$c->number}] {$c->url}\n";
}
```

### Methods

| Method | What it does |
| --- | --- |
| `scrape($url, $formats, $extra)` | One page, synchronously. Returns `Page`. |
| `startCrawl` / `crawl` / `getCrawl` / `waitForCrawl` / `cancelCrawl` / `crawlErrors` | The crawl lifecycle. `crawl()` is start + wait. |
| `startBatchScrape` / `batchScrape` | Up to 100 URLs through the job pipeline (follow with `waitForCrawl`). |
| `map($url, search:, limit:, maxDepth:)` | Discover a site's URLs. Returns `list<string>`. |
| `search($query, ...)` | Web search. Returns `SearchResponse`. |
| `research($query, depth:, output:, schema:)` | Cited answer. Returns `ResearchResult`. |
| `extract($urls, prompt:, schema:)` | Structured data across up to 10 URLs. |
| `llmsTxt($url, maxUrls:, fullText:)` | Generate `llms.txt` for a site. |
| `whoami()` / `orgId()` | The key's organization, scopes and plan. |
| `Client::pageMarkdown($page)` / `$page->markdown()` | Markdown of a page, whichever field carries it. |

All methods are synchronous and return readonly value objects (`Page`, `CrawlRecord`, `SearchResponse`, `ResearchResult`, `Citation`, ...) with `public readonly` properties for the stable fields. Every value object has `raw()`, the decoded array exactly as received, for fields the SDK does not model. Open-ended parts (`extract()`, `llmsTxt()`, `whoami()`) are plain arrays.

### Options

Common options are named arguments; anything else the API accepts goes in the `$extra` array (it wins over named options, `null` values are dropped):

```php
$page = $client->scrape(
    'https://example.com/pricing',
    formats: [
        Format::MARKDOWN,                                     // plain format
        Format::object(Format::JSON, prompt: 'Extract plans', // object format
            schema: ['type' => 'object', 'properties' => ['plans' => ['type' => 'array']]]),
    ],
    extra: ['only_main_content' => true, 'wait_for' => 1000],
);
```

`Format::names('markdown', 'links')` builds a list of plain formats; `Format::object($type, prompt:, schema:, question:, query:, maxResults:, extra:)` builds an object format.

### Waiting for crawls

`waitForCrawl()` polls with `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and follows the server's `Retry-After`. It throws `CrawlTimeoutException` after `$timeout` seconds (default 600); the crawl keeps running on the server.

```php
$id = $client->startCrawl('https://example.com', extra: ['limit' => 100]);
$record = $client->waitForCrawl($id, timeout: 300.0, pollInterval: 2.0);
$failed = $client->crawlErrors($id);   // failed pages and robots.txt-blocked URLs
```

## Configuration

```php
$client = new Client(
    apiKey: 'xd_...',                          // default: XDIVE_API_KEY, then CRAWLER_API_KEY
    baseUrl: Client::DEFAULT_BASE_URL,         // default: XDIVE_BASE_URL, then CRAWLER_BASE_URL
    orgId: 'org_123',                          // skips the one-time GET /whoami discovery
    maxRetries: 3,
    timeout: 120.0,                            // per attempt, default Guzzle client only
    httpClient: $psr18Client,                  // any Psr\Http\Client\ClientInterface
);
```

| Variable | Purpose |
| --- | --- |
| `XDIVE_API_KEY` (fallback `CRAWLER_API_KEY`) | API key |
| `XDIVE_BASE_URL` (fallback `CRAWLER_BASE_URL`) | API origin including `/api` |

The organization id is discovered once with `GET /whoami` and cached on the client instance. The client holds no other state, so create it once and reuse it.

To use another HTTP client, pass any PSR-18 client; request and stream factories default to `guzzlehttp/psr7` and can be injected too (`requestFactory:`, `streamFactory:`). When you inject a client, set its timeout yourself.

## Laravel

No service provider needed: bind the client in `app/Providers/AppServiceProvider.php`.

```php
use XDive\Client;

public function register(): void
{
    $this->app->singleton(Client::class, fn () => new Client(
        apiKey: config('services.xdive.key'),
        baseUrl: config('services.xdive.base_url'),   // null falls back to the default
        orgId: config('services.xdive.org_id'),
    ));
}
```

```php
// config/services.php
'xdive' => [
    'key' => env('XDIVE_API_KEY'),
    'base_url' => env('XDIVE_BASE_URL'),
    'org_id' => env('XDIVE_ORG_ID'),
],
```

Then type-hint `XDive\Client` in controllers, jobs and commands. To use Laravel's HTTP stack settings, pass an injected Guzzle client via `httpClient:`.

## Symfony

```yaml
# config/services.yaml
services:
    XDive\Client:
        arguments:
            $apiKey: '%env(XDIVE_API_KEY)%'
            $baseUrl: '%env(default::XDIVE_BASE_URL)%'
            $orgId: '%env(default::XDIVE_ORG_ID)%'
            $httpClient: '@psr18.http_client'   # optional: reuse Symfony's PSR-18 client
```

`default::VAR` resolves to `null` when the variable is unset, which makes the client use its defaults. `@psr18.http_client` requires `symfony/http-client` and `nyholm/psr7`; leave `$httpClient` out to use Guzzle.

## Error handling

Every exception the SDK throws implements the marker interface `XDive\Exception\XDiveException`.

```php
use XDive\Exception\ApiException;
use XDive\Exception\CrawlTimeoutException;
use XDive\Exception\QuotaExceededException;
use XDive\Exception\RateLimitedException;
use XDive\Exception\XDiveException;

try {
    $client->scrape('https://example.com');
} catch (RateLimitedException $e) {          // retries ran out on 429 / 503 SERVER_BUSY / 409 IDEMPOTENCY_KEY_IN_PROGRESS
    sleep((int) ceil($e->retryAfter()));
} catch (QuotaExceededException $e) {        // HTTP 402: plan quota exhausted
    // upgrade the plan
} catch (ApiException $e) {                  // any other error response
    error_log(sprintf('%d %s (%s) request=%s docs=%s',
        $e->status(), $e->errorCode(), $e->getMessage(), $e->requestId(), $e->docsUrl()));
    if ($e->retryable()) { /* the server says the same request may succeed later */ }
} catch (XDiveException $e) {                // TransportException, CrawlTimeoutException, ConfigurationException ...
}
```

`ApiException extends \RuntimeException` and exposes `status()`, `errorCode()` (null for older error bodies), `retryable()`, `requestId()` (the `X-Request-ID` header) and `docsUrl()`. `RateLimitedException` adds `retryAfter()` in float seconds; `CrawlTimeoutException` (also an `ApiException`, status 408) adds `crawlId()`. `TransportException` means no HTTP response arrived after all retries; the PSR-18 exception is `getPrevious()`.

## Retries and idempotency

The client retries, up to `maxRetries` times (default 3), on:

- `429`, `503` with code `SERVER_BUSY`, and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS`, waiting the `Retry-After` seconds (default 1, fractions allowed);
- transport errors and timeouts for GET and POST, with backoff 1s, 2s, 4s, capped at 8s.

Every other status is never retried. Every POST carries an `Idempotency-Key` header (a random UUID v4) generated once per call and reused by every retry, so a retried scrape or crawl never runs twice. Pass your own to make a call safe to repeat from your side too:

```php
$client->startCrawl('https://example.com', idempotencyKey: 'nightly-2026-10-18');
```

Calls with the same key return the first call's response. A new call without a key gets a new key.

## Testing your code

`Client` accepts any PSR-18 client, so tests can inject a mock (for example Guzzle's `MockHandler`). The constructor also takes `sleeper:` and `clock:` callables to make retries and `waitForCrawl()` run without real delays.

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-php · All docs for agents: https://xdive.dev/llms.txt


---

# Ruby SDK

Official Ruby client for the [XDive](https://xdive.dev) web-data API: scrape pages to Markdown, crawl sites, search the web and get cited research answers.

- Ruby 3.0+, standard library only (`net/http`, `json`, `securerandom`, `uri`), no Rails dependency (works in Rails).
- Thread-safe: one client can be shared by every thread; each call uses its own connection.
- Automatic retries with `Retry-After`, and an `Idempotency-Key` on every POST so a retried call never runs (or bills) twice.

## Install

```sh
gem install xdive
```

or in your `Gemfile`:

```ruby
gem "xdive"
```

## Quickstart

```ruby
require "xdive"

client = XDive::Client.new(api_key: ENV.fetch("XDIVE_API_KEY"))
# or simply: client = XDive.client   (reads XDIVE_API_KEY / XDIVE_BASE_URL)

# Scrape one page
page = client.scrape("https://example.com", formats: %w[markdown links])
puts page.title
puts page.markdown          # also XDive.page_markdown(page)

# Crawl a site and wait for it (start_crawl + wait_for_crawl)
record = client.crawl("https://docs.example.com", limit: 50, formats: ["markdown"])
record.results.each { |p| puts "#{p.url}: #{p.markdown.length} chars" } if record.done?

# Search the web
hits = client.search("ruby 4.0 release notes", limit: 5, include_answer: true)
hits["results"].each { |r| puts "#{r['title']} - #{r['url']}" }

# Research: a cited answer
result = client.research("What changed in Ruby 4.0?", depth: "quick")
puts result.answer
result.citations.each { |c| puts "[#{c['number']}] #{c['url']}" }
```

### All methods

| Method | Returns |
| --- | --- |
| `scrape(url, formats:, **opts)` | `XDive::Page` |
| `start_crawl(url, formats:, **opts)` / `crawl(url, timeout: 600, poll_interval: 2, ...)` | job id / `XDive::CrawlRecord` |
| `get_crawl(id)`, `wait_for_crawl(id, timeout:, poll_interval:)` | `XDive::CrawlRecord` (`#done?`, `#results`) |
| `cancel_crawl(id)`, `crawl_errors(id)` | `nil`, `Hash` |
| `start_batch_scrape(urls, ...)` / `batch_scrape(urls, ...)` | job id / `XDive::CrawlRecord` |
| `map(url, search:, limit:, max_depth:)` | `Array<String>` |
| `search(query, **opts)` | `Hash` |
| `research(query, depth:, output:, schema:, no_cache:)` | `XDive::ResearchResult` (`#answer`, `#citations`) |
| `extract(urls, prompt:, schema:)`, `llms_txt(url, max_urls:, full_text:)`, `whoami` | `Hash` |

Results are plain string-keyed Hashes; `Page`, `CrawlRecord` and `ResearchResult` are thin wrappers that expose the complete response as `#raw` / `#to_h` and accept `result[:key]`. Request fields the gem does not model pass through as extra keyword arguments or a trailing Hash: `client.scrape(url, only_main_content: true)`, `client.scrape(url, { "wait_for" => 500 })`. `nil` values are dropped. `formats:` takes names (`"markdown"`, `:links`) or option hashes (`{ type: "json", schema: {...} }`).

## Rails

```ruby
# config/initializers/xdive.rb
XDive.configure(
  api_key: Rails.application.credentials.dig(:xdive, :api_key),
  timeout: 60,
  max_retries: 3
)
```

Then use the shared, thread-safe client anywhere: `XDive.client.scrape("https://example.com").markdown`. For long crawls prefer a background job (`start_crawl` in the request, `wait_for_crawl` or `get_crawl` in the job).

## Errors

Every failed call raises a subclass of `XDive::Error`:

```ruby
begin
  client.scrape("https://example.com")
rescue XDive::RateLimitedError => e
  sleep e.retry_after            # still throttled after max_retries
rescue XDive::QuotaExceededError
  # HTTP 402: plan quota used up
rescue XDive::CrawlTimeoutError => e
  e.crawl_id                     # wait_for_crawl gave up; the crawl keeps running
rescue XDive::ConnectionError
  # DNS / refused / timeout / TLS, after retries (see #cause)
rescue XDive::Error => e
  warn "#{e.status} #{e.code}: #{e.message} (request #{e.request_id}) #{e.docs_url}"
  retry if e.retryable?
end
```

`#status`, `#code`, `#retryable?`, `#request_id` (the `X-Request-ID` header) and `#docs_url` come from the API's error body `{"success":false,"code":"..","error":"..","retryable":bool,"docs_url":".."}`; older bodies with only `error` leave `#code` empty.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS` are retried after `Retry-After` seconds (default 1, fractional allowed), up to `max_retries` (default 3), then `RateLimitedError` is raised. Other statuses are never retried.
- Transport errors and timeouts on GET and POST are retried with a 1s, 2s, 4s... backoff (cap 8s), then `ConnectionError` is raised.
- Every POST carries an `Idempotency-Key` (a UUID generated once per call and reused by every retry of that call). Pass `idempotency_key: "my-key"` to make a call safe to repeat across processes.
- `wait_for_crawl` sends `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and follows the server's `Retry-After`.

## Configuration

| Option | Env var (fallback) | Default |
| --- | --- | --- |
| `api_key:` | `XDIVE_API_KEY` (`CRAWLER_API_KEY`) | required |
| `base_url:` | `XDIVE_BASE_URL` (`CRAWLER_BASE_URL`) | `XDive::DEFAULT_BASE_URL` |
| `org_id:` | none | discovered once via `GET /whoami`, cached |
| `max_retries:` | none | `3` |
| `timeout:` | none | `120` seconds (connect, read and write) |

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-ruby · All docs for agents: https://xdive.dev/llms.txt


---

# Rust SDK

The official Rust client for the [XDive](https://xdive.dev) web
data API: scrape, crawl, map, search, research and extract. Async, built on
`reqwest` with `rustls` (no OpenSSL).

## Install

```sh
cargo add xdive
cargo add tokio --features macros,rt-multi-thread
```

## Configuration

| Setting  | Builder method | Environment variable                     |
| -------- | -------------- | ---------------------------------------- |
| API key  | `api_key`      | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`)   |
| Base URL | `base_url`     | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`) |

Builder values win over the environment. The default base URL is
`xdive::DEFAULT_BASE_URL`. One API key identifies one organization; the client
looks the organization id up once with `GET /whoami` and caches it (skip the
lookup with `.org_id(..)`). Other builder methods: `http_client`, `max_retries`
(default 3) and `timeout` (per attempt, default 120 seconds).

```rust,no_run
use std::time::Duration;
use xdive::Client;

// From the environment (XDIVE_API_KEY, optionally XDIVE_BASE_URL)...
let client = Client::from_env()?;

// ...or explicitly.
let client = Client::builder()
    .api_key("ck_live_...")
    .max_retries(5)
    .timeout(Duration::from_secs(60))
    .build()?;
```

## Quickstart

```rust,no_run
use xdive::{Client, CrawlOptions, ResearchOptions, ScrapeOptions, SearchOptions};

#[tokio::main]
async fn main() -> Result<(), xdive::Error> {
    let client = Client::from_env()?;

    // Scrape one page.
    let page = client
        .scrape("https://example.com", ScrapeOptions::new().formats(["markdown"]))
        .await?;
    println!("{}", xdive::page_markdown(&page));

    // Crawl a site and wait for it to finish.
    let record = client
        .crawl(
            "https://example.com",
            CrawlOptions::new().formats(["markdown"]).field("limit", 20),
        )
        .await?;
    println!("{} {}", record.status, record.results.len());

    // Search the web.
    let found = client
        .search("rust async runtimes", SearchOptions::new().limit(5))
        .await?;
    for hit in &found.results {
        println!("{} {}", hit.title, hit.url);
    }

    // Research a question with verified citations.
    let answer = client
        .research("What changed in Rust 1.80?", ResearchOptions::new())
        .await?;
    println!("{} ({})", answer.answer.unwrap_or_default(), answer.confidence);
    for c in &answer.citations {
        println!("[{}] {}", c.number, c.url);
    }
    Ok(())
}
```

Long-running jobs can be started and followed separately:

```rust,no_run
use std::time::Duration;
use xdive::{Client, CrawlOptions, Error, WaitOptions};

let id = client.start_crawl("https://example.com", CrawlOptions::new()).await?;
match client
    .wait_for_crawl(&id, WaitOptions::new().timeout(Duration::from_secs(60)))
    .await
{
    Ok(record) => println!("{} pages", record.results.len()),
    // The crawl keeps running on the server; check on it later.
    Err(Error::CrawlTimeout { crawl_id }) => println!("still running: {crawl_id}"),
    Err(e) => return Err(e),
}
```

Other methods: `start_batch_scrape` / `batch_scrape`, `get_crawl`,
`cancel_crawl`, `crawl_errors`, `map`, `extract`, `llms_txt` and `whoami`.
Request options take an `extra(..)` / `field(key, value)` escape hatch for API
fields the SDK does not model, and results keep unknown fields in `extra`.

## Output formats

Plain formats are names: `.formats(["markdown", "links"])`. Formats with
options are `Format` values:

```rust
use xdive::{Format, ScrapeOptions};

let opts = ScrapeOptions::new().formats([
    Format::markdown(),
    Format::json().prompt("Extract the product name and price"),
]);
```

## Error handling

Every method returns `Result<T, xdive::Error>`.

```rust,no_run
use xdive::{Client, Error, ScrapeOptions};

match client.scrape("https://example.com", ScrapeOptions::new()).await {
    Ok(page) => println!("{}", page.title.unwrap_or_default()),
    Err(Error::RateLimited { error, retry_after }) => {
        eprintln!("slow down ({}), retry in {retry_after:?}", error.message);
    }
    Err(Error::QuotaExceeded(e)) => eprintln!("plan quota used up: {}", e.message),
    Err(Error::Api(e)) => eprintln!(
        "API error {} {:?}: {} (request {:?})",
        e.status, e.code, e.message, e.request_id
    ),
    Err(Error::Transport(e)) => eprintln!("network problem: {e}"),
    Err(e) => eprintln!("{e}"),
}
```

`Error::is_retryable()` tells whether repeating the request later may work and
`Error::status()` returns the HTTP status when the error came from a response.
`ApiError` carries `status`, `code`, `message`, `retryable`, `request_id` and
`docs_url`.

## Retries and idempotency

The client retries, up to `max_retries` times (default 3):

- HTTP 429, 503 with code `SERVER_BUSY` and 409 with code
  `IDEMPOTENCY_KEY_IN_PROGRESS`, after waiting the `Retry-After` header
  (default 1 second, fractional seconds allowed). When the budget runs out the
  result is `Error::RateLimited`.
- Transport errors and timeouts, for GET and POST, with a backoff of 1s, 2s,
  4s, ... capped at 8s.

Other statuses are never retried. Every POST carries an `Idempotency-Key` header
(a random UUID v4, generated once per call and reused by each retry), so a
retry after a timeout cannot run the work twice. Pass your own key to make a
call safe to repeat across processes:

```rust,no_run
use xdive::{Client, CrawlOptions};

let id = client
    .start_crawl(
        "https://example.com",
        CrawlOptions::new().idempotency_key("nightly-2026-10-18"),
    )
    .await?;
```

Dropping a future (for example with `tokio::time::timeout` or `select!`) cancels
the call cleanly, including any retry wait.

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-rust · All docs for agents: https://xdive.dev/llms.txt


---

# Vite plugin

`vite-plugin-xdive` turns web pages into LLM-ready Markdown while you build, so your app can import them like any other module. Use it to ship docs, pricing or policy pages that are kept in step with a source site, or to generate an `llms.txt` for your own site. It uses your API key and the [scrape](https://xdive.dev/docs/scrape.md) endpoint, and caches every page on disk.

## Install

```sh
npm install -D vite-plugin-xdive
pnpm add -D vite-plugin-xdive
yarn add -D vite-plugin-xdive
bun add -d vite-plugin-xdive
```

The plugin needs Vite 5, 6 or 7 and Node 18 or newer. Set your API key in the environment:

```sh
export XDIVE_API_KEY=ck_live_...
```

## Usage

```ts
// vite.config.ts
import { defineConfig } from "vite";
import xdive from "vite-plugin-xdive";

export default defineConfig({
  plugins: [
    xdive({
      pages: [
        { id: "pricing", url: "https://example.com/pricing" },
        { url: "https://example.com/docs/intro" }, // id: example-com-docs-intro
      ],
      llmsTxt: { title: "Example", summary: "Docs and pricing for Example." },
    }),
  ],
});
```

For TypeScript, add once (for example in `src/env.d.ts`):

```ts
/// <reference types="vite-plugin-xdive/client" />
```

React:

```tsx
import { pages } from "virtual:xdive";

export function Pricing() {
  return <pre>{pages.pricing.markdown}</pre>;
}
```

Vue:

```vue
<script setup lang="ts">
import { pages } from "virtual:xdive";
const { title, markdown } = pages.pricing;
</script>
<template><h1>{{ title }}</h1><pre>{{ markdown }}</pre></template>
```

Each page is `{ url, title, markdown, fetchedAt }`. `pages` is also the default export.

## Options

| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `pages` | `{ id?, url, formats?, maxAgeMs? }[]` | required | Pages to fetch. `id` defaults to a slug of the URL; `markdown` is always requested; a cached copy younger than `maxAgeMs` is used without calling the API. |
| `apiKey` | `string` | env `XDIVE_API_KEY`, then `CRAWLER_API_KEY` | Sent as `X-API-Key`. Never written to the cache or logs. |
| `baseUrl` | `string` | env `XDIVE_BASE_URL`, then `CRAWLER_BASE_URL`, then the public API | API origin including `/api`. |
| `cacheDir` | `string` | `node_modules/.cache/xdive` | Relative to the Vite root. |
| `offline` | `boolean` | `false` | Use only the cache; no API calls, no key needed. A page with no cached copy is an error in build and omitted with a warning in dev. |
| `failOnError` | `boolean` | unset | See below. |
| `llmsTxt` | `{ title, summary?, outFile? }` | off | Emit `llms.txt` (llmstxt.org format) and `<id>.md` for every page into the build output. `outFile` defaults to `llms.txt`. |

`.env` files are read too (`XDIVE_API_KEY` in `.env.local` works).

## Caching and failures

One JSON file per page is kept in the cache directory.

- `vite build` always refreshes every page (except pages whose cached copy is younger than `maxAgeMs`).
- `vite dev` serves the cached copy instantly and refreshes it in the background; if the content changed, the virtual module is invalidated and the page reloads. A page with no cache is fetched before the first load.
- If a fetch fails, `failOnError` decides:
  - unset (default): the cached copy is used with a warning. With no cached copy, the build fails and `vite dev` omits the page with a warning.
  - `false`: never fails; cached copy with a warning, else the page is omitted with a warning.
  - `true`: any failed page fails the build or dev start; the cache is not used as a fallback.
- A missing API key always fails (unless `offline`), naming `XDIVE_API_KEY` and `CRAWLER_API_KEY`.
- Quota errors (402) and other HTTP errors include the API's message and `code`.
- 429 and 503 `SERVER_BUSY` are retried honoring `Retry-After`, reusing one `Idempotency-Key` so a retry never runs or bills a request twice.

## CI

Store the key as a secret and expose it as `XDIVE_API_KEY`:

```yaml
- run: pnpm build
  env:
    XDIVE_API_KEY: ${{ secrets.XDIVE_API_KEY }}
```

Cache `node_modules/.cache/xdive` between runs so a transient API outage does not break the build.

## See also

- [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md), if you want to call the API from your own code.
- [Scrape](https://xdive.dev/docs/scrape.md), the endpoint the plugin calls.
- [Errors and limits](https://xdive.dev/docs/errors.md), for quotas and retries.

---

Source: https://xdive.dev/docs/vite-plugin · All docs for agents: https://xdive.dev/llms.txt


---

# Scrape

Fetch one URL and get it back as clean Markdown, HTML, plain text, links, images, a screenshot or structured data. Scrape is synchronous: the response carries the page. Use it when you need a single page now; use [Batch scrape](https://xdive.dev/docs/batch-scrape.md) for lists of URLs and [Crawl](https://xdive.dev/docs/crawl.md) to follow links across a site.

`POST https://xdive.dev/api/orgs/{id}/scrape`

Requires an API key with the `scrape` scope. A scrape times out after 30 seconds, or 3 minutes when `audio` is requested; [browser-backed scrapes](#browser-actions) allow 45 seconds per step. HTML, PDF, DOCX, XLSX and plain-text responses up to 50 MB are converted.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "links"],
    "max_age": 3600000
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape(
    "https://example.com/pricing",
    formats=["markdown", "links"],
    max_age=3_600_000,
)
print(page["artifacts"]["markdown"]["content"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com/pricing", {
  formats: ["markdown", "links"],
  max_age: 3_600_000,
});
console.log(page.artifacts?.markdown?.content);
```

The SDKs return the `data` object of the response. They look up your organization id once through `GET /whoami`, so you only configure the key.

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Absolute `http` or `https` URL to fetch. |
| `formats` | array | `["markdown"]` | Representations to return. Each entry is a format name or an object with per-format options. See [Formats](#formats). |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers, footers and sidebars. `false` converts the whole page. |
| `respect_robots` | boolean | `true` | Refuse URLs that the site's robots.txt disallows for the crawler. |
| `max_age` | integer | `0` | Freshness window in milliseconds. A cached copy younger than this is served without a network fetch. `0` uses the server's default window. See [Caching](#caching-and-max_age). |
| `max_chars` | integer | `0` | Return at most this many characters of Markdown, cut at a paragraph break. `0` means no limit; otherwise at least 500. See [Long pages](#long-pages). |
| `offset` | integer | `0` | Continue a long page from the previous response's `next_offset`. |
| `store_in_cache` | boolean | `true` | `false` bypasses the page cache for this request: the page is fetched live and not stored. |
| `headers` | object | none | Extra request headers for the page fetch, such as `Accept-Language`, a mobile `User-Agent` or `Cookie`. See [Custom headers](#custom-headers). |
| `actions` | array | none | Browser steps to run before the content is captured (click, fill, press, scroll, wait). At most 15. See [Browser actions](#browser-actions). |
| `wait_for` | integer | `0` | Milliseconds to wait after the page loads and the actions run, before capture. 0 to 10000. Uses the browser. |
| `full_page_screenshot` | boolean | `false` | With the `screenshot` format, capture the full scrollable page instead of the viewport. |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses with typed tokens. See [Privacy controls](#privacy-controls). |
| `zero_data_retention` | boolean | `false` | Do not write the page to the cache and mark the response `no-store`. |
| `lockdown` | boolean | `false` | Serve only an already-cached copy and make no outbound page fetch. Implies `zero_data_retention`. |

## Formats

Pass format names as strings, or objects when a format takes options. Each format may appear once. Names are case-insensitive and a few aliases are accepted (`md`, `txt`, `raw_html`, `clean_html`, `summarize`, `answer`, `screenshots`, `mp3`); the response always uses the canonical name.

| Format | Returned in the artifact | Notes |
|---|---|---|
| `markdown` | `content` | Clean Markdown. The default. |
| `text` | `content` | Plain text. |
| `html` | `content` | Clean HTML rendered from the parsed page (main content when `extract` is on). |
| `rawHtml` | `content` | The HTML as fetched. |
| `links` | `links`: list of `url` and `text` | Every link on the page, whatever `extract` is set to. |
| `images` | `images`: list of `url`, `alt`, `title` | Every image on the page. |
| `branding` | `branding` | Observed identity signals: `name`, `description`, `logo`, `favicon`, `colors`, `fonts`, `css_variables`. Absent values stay absent. |
| `summary` | `content` | A short summary: 3 to 6 sentences written by your LLM when one is configured, otherwise 5 sentences extracted from the page. Accepts `prompt` for extra instructions. |
| `question` | `content` | Answers `question` (or `prompt`) from the page only. Uses your LLM when configured, otherwise the most relevant sentences. |
| `highlights` | `highlights`: list of `text` and `score` | Sentences most relevant to `query` (or `prompt`), best first. No LLM. `max_results` sets the count (default 5). |
| `json` | `json` | Structured data from `prompt` and/or a JSON Schema in `schema`. Requires an LLM. |
| `product` | `json` | Product fields (title, brand, price, currency, availability, sku, variants…) from a built-in schema. Requires an LLM. |
| `menu` | `json` | Restaurant menu sections and items from a built-in schema. Requires an LLM. |
| `screenshot` | `content` | A PNG as a `data:image/png;base64,…` URL, at most 8 MiB. Runs in the browser. See [Screenshots](#screenshots). |
| `audio` | `url` | The audio track of an HTTPS YouTube or Vimeo URL as MP3, up to 100 MB. See [Audio](#audio). |

The LLM is the model configured for your organization in AI settings (or the platform's). Without one, `json`, `product` and `menu` fail with `501`. Text sent to the model is capped at 120,000 characters.

### Format options

| Option | Type | Used by | Description |
|---|---|---|---|
| `type` | string | all | The format name. |
| `prompt` | string | `json`, `summary`, `question`, `highlights` | Instructions for the format. For `question` and `highlights` it stands in for `question` and `query`. |
| `schema` | object | `json` | A JSON Schema for the result. Must be valid JSON. |
| `question` | string | `question` | The question to answer. `question` or `prompt` is required. |
| `query` | string | `highlights` | What the highlights should be relevant to. Without it, the page's most representative sentences are returned. |
| `max_results` | integer | `highlights` | Number of highlights, 0 to 50. |

```json
{
  "url": "https://example.com/product/espresso-grinder",
  "formats": [
    "markdown",
    { "type": "json", "prompt": "Extract the product", "schema": {
      "type": "object",
      "properties": { "name": { "type": "string" }, "price": { "type": "number" } }
    } },
    { "type": "highlights", "query": "warranty", "max_results": 3 }
  ]
}
```

### Screenshots

Requesting `screenshot` routes the scrape through an isolated, short-lived browser session instead of the standard fetch. The other formats you ask for are rendered from the same final DOM. Add `full_page_screenshot: true` for the whole page.

### Audio

`audio` extracts the audio of an HTTPS YouTube (`youtube.com`, `youtu.be`) or Vimeo URL. The artifact's `url` is a signed link to the MP3 that expires after one hour. Audio needs object storage on the server, cannot be combined with `zero_data_retention` or `lockdown`, and is not available on crawls or batch scrapes. If extraction fails the request returns `422`.

## Main-content extraction

With `extract` on (the default), the page's main article is found and navigation, headers, footers and sidebars are dropped before conversion, so `markdown`, `text` and `html` hold the content a reader came for. Set `extract: false` to convert the whole document. Metadata (`title`, `description`, `lang`…), `links` and `images` always describe the whole page.

## Caching and max_age

Pages are cached across the platform. On a fresh hit the response is served without a network fetch and the page reports `fetch_route: "cache"` with `fetch_cost_units: 0`.

- `max_age` sets how old a cached copy may be, in milliseconds. An older copy is fetched again from the site, and the fresh response replaces it. A window longer than the server default accepts older copies for as long as the cache still holds them.
- `store_in_cache: false` skips the cache in both directions for this request.
- `zero_data_retention` and `lockdown` never write to the cache.
- Responses fetched with custom `headers` are cached separately from header-less fetches, so a page fetched with your cookies is never served to a request without them.

## Robots.txt

By default the crawler checks the site's robots.txt and refuses disallowed URLs with `403` and `disallowed by robots.txt`. Set `respect_robots: false` to skip the check when you have the right to fetch the page.

## Long pages

Every Markdown artifact reports `estimated_tokens`, about four characters per token, so an agent knows what a page costs to read before it reads it. To read a long page in parts, pass `max_chars`. The artifact then carries `total_chars` and, while more remains, `next_offset`. Send the same request again with `offset` set to `next_offset` to get the next part. A continuation reuses the copy fetched for the first part (up to 10 minutes old), so all parts come from the same fetch. Each request is metered as one page.

```json
{ "url": "https://go.dev/doc/effective_go", "max_chars": 4000 }
```

```json
"markdown": { "content": "# Effective Go …", "estimated_tokens": 915, "total_chars": 102176, "next_offset": 3661 }
```

## JavaScript pages and warnings

A plain scrape starts with a fast HTTP fetch. When the result is a JavaScript application shell or loses most of the page's text, the server renders the page again in a browser and returns that instead; `fetch_route` is then `browser` and `fetch_reason` says why. The browser time is metered as browser seconds, on top of the page.

When the content still looks incomplete, for example because the browser quota is used up or the request sends custom `headers` (a browser cannot send them), the page succeeds with a `warnings` array explaining it. Retry with `wait_for` to give a slow page more time to render, or use what you got.

## Custom headers

`headers` adds up to 10 request headers to the page fetch. Names are 1 to 128 characters, values at most 2048 and without line breaks. `Host`, `Content-Length`, `Transfer-Encoding`, `Connection`, `Upgrade`, `TE`, `Trailer`, `Proxy-Authorization` and any `X-Forwarded-*` header are rejected. A `User-Agent` you set replaces the crawler's. Headers apply to the HTTP fetch; browser-backed scrapes do not send them.

## Browser actions

`actions`, `wait_for` and the `screenshot` format run the scrape in an isolated, ephemeral browser session: it opens `url`, runs your actions in order, waits `wait_for` milliseconds, captures the DOM and renders the requested formats from it. Each action has 45 seconds to complete.

| Action | Fields | Description |
|---|---|---|
| `click` | `selector` | Click the first element matching a CSS selector. |
| `fill` | `selector`, `value` | Wait for the element to be visible, focus it and set its value. |
| `press` | `key` | Press one of `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `Space`. |
| `scroll` | `x`, `y` | Scroll the window by this many pixels (each between -100000 and 100000). |
| `wait` | `selector` or `milliseconds` | Wait until the selector is visible, or sleep 1 to 10000 milliseconds. |

Selectors are at most 1000 characters and values at most 10000. `goto`, `extract` and `screenshot` actions are not accepted on scrape: the scrape navigates to `url` itself, and you request content and screenshots through `formats`.

```json
{
  "url": "https://example.com/search",
  "actions": [
    { "type": "fill", "selector": "input[name=q]", "value": "espresso" },
    { "type": "press", "key": "Enter" },
    { "type": "wait", "selector": ".results" }
  ],
  "wait_for": 1000,
  "formats": ["markdown", "screenshot"]
}
```

Browser-backed scrapes are metered as browser seconds (rounded up, at least one) against your plan's browser quota instead of pages. They need browser automation on the server (`501` otherwise) and a free browser session (`409` when the server's sessions, or your plan's, are all in use). `respect_robots` applies to the starting URL (on by default; a disallowed URL returns `403`). `headers` cannot be sent by the browser and are rejected with `400`; `max_age` and `store_in_cache` do not apply, since a browser scrape always renders live. They cannot be combined with `lockdown`. They are not available through `GET`.

## Privacy controls

**`redact_pii`** deterministically replaces machine-recognizable identifiers in the returned title, description and text artifacts (including `json` and `highlights`) with typed tokens: `[REDACTED:email]`, `[REDACTED:phone]`, `[REDACTED:card]` (Luhn-checked), `[REDACTED:ssn]`, `[REDACTED:iban]` and `[REDACTED:ip]`. Names and street addresses are not detected. Link and image URLs are not changed.

**`zero_data_retention`** keeps the page out of the cache, sets `Cache-Control: no-store, private` and `Pragma: no-cache`, and the audit log entry for a standard scrape omits the URL. Formats that use an LLM still send the page text to your configured model.

**`lockdown`** serves the page only if it is already cached, whatever its age. No DNS lookup, robots.txt fetch, HTTP request or browser session is made. A cache miss returns `404` with `SCRAPE_LOCKDOWN_CACHE_MISS`.

> **Warning:**
>
> Lockdown makes no model calls either. Request only formats rendered from the page (`markdown`, `text`, `html`, `rawHtml`, `links`, `images`, `branding`, `highlights`); asking for `json`, `product` or `menu`, or for `summary` or `question` when a model is configured, returns `400` before any work is done.

A platform administrator can make zero data retention or lockdown the default for every request. A successful POST reports the policy that applied in `security`, and both POST and GET responses carry two headers:

| Header | Values |
|---|---|
| `X-Data-Retention` | `none` with zero data retention, otherwise `standard` |
| `X-Network-Policy` | `cache_only` under lockdown, otherwise `live` |

## Response

```json
{
  "success": true,
  "data": {
    "url": "https://example.com/pricing",
    "title": "Pricing",
    "description": "Plans for teams of every size.",
    "lang": "en",
    "depth": 0,
    "status_code": 200,
    "word_count": 412,
    "fetched_at": "2026-10-15T09:21:44.118302Z",
    "fetch_route": "impersonate",
    "fetch_attempts": 2,
    "fetch_cost_units": 3,
    "fetch_reason": "blocked or transient HTTP status 403",
    "artifacts": {
      "markdown": {
        "media_type": "text/markdown; charset=utf-8",
        "content": "# Pricing\n\nStart free, then pay as you grow...",
        "bytes": 2841,
        "quality": { "passed": true, "score": 100 }
      },
      "links": {
        "media_type": "application/json",
        "links": [{ "url": "https://example.com/docs", "text": "Docs" }],
        "bytes": 96,
        "quality": { "passed": true, "score": 100 }
      }
    },
    "links": ["https://example.com/docs", "https://example.com/signup"]
  },
  "security": {
    "zero_data_retention": false,
    "network_policy": "live",
    "cache_state": "eligible",
    "outbound_permitted": true
  }
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true` on success. |
| `data` | object | The page. |
| `security` | object | The data policy that applied to this request. |

### The page object

| Field | Type | Description |
|---|---|---|
| `url` | string | Final URL after redirects. |
| `title`, `description`, `lang`, `canonical`, `image`, `site_name` | string | Page metadata, when present. |
| `depth` | integer | Always `0` for a scrape. |
| `status_code` | integer | HTTP status of the fetch. |
| `word_count` | integer | Words in the page's Markdown. |
| `fetched_at` | string | When the content was fetched (RFC 3339). |
| `fetch_route`, `fetch_attempts`, `fetch_cost_units`, `fetch_reason` | | How the page was fetched. See [Fetch telemetry](#fetch-telemetry). |
| `artifacts` | object | One artifact per requested format, keyed by format name. Only requested formats are returned. |
| `links` | array of strings | URLs of the links found on the page. |

Each artifact has `media_type`, `bytes`, a `quality` report (`passed`, `score` from 0 to 100, optional `warnings`) and the field for its format: `content`, `json`, `links`, `images`, `highlights`, `branding` or `url`.

### Fetch telemetry

Each page reports how it was fetched. A plain HTTP request is tried first; on a block, challenge or bot-filter response the fetch escalates to a Chrome TLS fingerprint, and on servers with an egress proxy, sites that refuse the server's network are retried through the proxy.

| Field | Description |
|---|---|
| `fetch_route` | The route that produced the content: `cache`, `http`, `impersonate`, `browser` or `proxy`. |
| `fetch_attempts` | Network attempts made, across routes. `0` for a cache hit. |
| `fetch_cost_units` | Relative cost of the attempts: 1 per HTTP attempt, 2 per impersonated attempt, 20 per browser render, plus 5 for the egress proxy. `0` for a cache hit. It is telemetry, not your bill: a scrape is metered as one page. |
| `fetch_reason` | Why the fetch escalated or where it came from, for example `blocked or transient HTTP status 403`, `JavaScript application shell detected` or `fresh response cache hit`. Empty when the first HTTP attempt succeeded. |

### The security object

| Field | Values |
|---|---|
| `zero_data_retention` | Whether zero data retention applied. |
| `network_policy` | `live`, or `cache_only` under lockdown. |
| `cache_state` | `eligible` (the page may be cached), `write_disabled` (it was not stored), or `hit` (lockdown served the cache). |
| `outbound_permitted` | `false` under lockdown. |

## Government and banking sites

Many government and banking sites refuse every connection from data-centre networks, so no fetch strategy reaches them from the crawler's own address. XDive can route those requests through a residential proxy. The proxy is paid per page, so it is off for every organization until the platform administrator enables it for yours, with a monthly allowance of proxied pages (200 by default). Your current allowance and use are in the `proxy` object of [`GET /orgs/{id}/usage`](https://xdive.dev/docs/usage.md).

With access enabled, scrapes, crawls and batch scrapes go direct first and use the proxy only for sites that cannot be reached or stay blocked. Proxied pages report `fetch_route: "proxy"`, and each one counts once against the allowance. When the allowance is used up, fetches go direct only until the next calendar month.

Two errors are specific to these sites:

- **`403 PROXY_NOT_ENABLED`**: the site blocks data-centre networks, and your organization has no proxy access or has used its allowance. Ask the platform administrator to enable the proxy, or to raise the allowance.
- **`403 SITE_RESTRICTED`**: the proxy is enabled for your organization, but the proxy provider refused the destination. Providers typically refuse government and banking sites until the platform's proxy account is identity-verified. Retrying will not help.

In crawl and batch results, the same codes appear as the failed page's `error_code`.

## GET (Markdown only)

`GET https://xdive.dev/api/orgs/{id}/scrape`

The original endpoint returns the page's Markdown as `text/markdown` instead of JSON. It accepts the query parameters `url`, `respect_robots`, `zero_data_retention` and `lockdown`, and the same errors as POST. Formats, actions, headers and `max_age` need POST.

```bash
curl "https://xdive.dev/api/orgs/$ORG_ID/scrape?url=https://example.com" \
  -H "X-API-Key: $XDIVE_API_KEY"
```

## Errors

Errors have an `error` message; the ones below that carry a `code` also have `"success": false`.

| Status | Code | When |
|---|---|---|
| `400` | | Missing `url`, an unknown or repeated format, invalid format options, an invalid header, a negative `max_age`, more than 15 actions, `wait_for` out of range, or an option combination that is not allowed. |
| `402` | | The monthly page quota (or browser quota, for browser scrapes) is used up. |
| `403` | | The key lacks the `scrape` scope, or robots.txt disallows the URL. |
| `403` | `PROXY_NOT_ENABLED` | The site blocks data-centre networks and your organization has no residential proxy access, or has used its monthly proxy allowance. See [Government and banking sites](#government-and-banking-sites). |
| `403` | `SITE_RESTRICTED` | Your organization may use the residential proxy, but the proxy provider refused this destination, which is common for government and banking sites. Retrying will not help. |
| `404` | `SCRAPE_LOCKDOWN_CACHE_MISS` | Lockdown is on and the page is not cached. |
| `409` | | No browser session is free for a browser-backed scrape. |
| `422` | | Audio extraction failed or the URL is not a supported video. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `501` | | A requested format needs an LLM that is not configured, or the server has no browser automation. |
| `502` | | The page could not be fetched or converted (`scrape failed: …`), or an enrichment format failed. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors, rate limits and retries.

## Related

- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) to scrape up to 100 URLs as one job.
- [Crawl](https://xdive.dev/docs/crawl.md) to follow links across a site.
- [Extract](https://xdive.dev/docs/extract.md) for structured data across several pages.
- [Usage and billing](https://xdive.dev/docs/usage.md) for what is metered.

---

Source: https://xdive.dev/docs/scrape · All docs for agents: https://xdive.dev/llms.txt


---

# Crawl

Start at one URL, follow its links, and convert every page reached. A crawl is asynchronous: the request returns a crawl id at once, a worker does the fetching, and you poll the crawl or receive a webhook when it finishes. Use it to ingest a documentation site, a blog or a section of a site; use [Map](https://xdive.dev/docs/map.md) when you only need the URLs.

`POST https://xdive.dev/api/orgs/{id}/crawl`

Requires an API key with the `crawl` scope. Starting a crawl counts against your plan's request rate limit; polling it does not.

**cURL**

```bash
# Start the crawl
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "max_pages": 50,
    "include_paths": ["^/guides/"],
    "formats": ["markdown"]
  }'
# {"id":"crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d","status":"queued"}

# Poll until status is success, error or cancelled
curl "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY"
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY

# crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
result = xdive.crawl(
    "https://docs.example.com",
    max_depth=2,
    max_pages=50,
    include_paths=["^/guides/"],
    formats=["markdown"],
)
for page in result["results"]:
    print(page["url"], page.get("error", ""))

# Or start it and poll yourself:
crawl_id = xdive.start_crawl("https://docs.example.com", max_pages=50)
record = xdive.wait_for_crawl(crawl_id, timeout=600)
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY

// crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
const result = await xdive.crawl("https://docs.example.com", {
  max_depth: 2,
  max_pages: 50,
  include_paths: ["^/guides/"],
  formats: ["markdown"],
});
for (const page of result.results ?? []) console.log(page.url, page.error ?? "");

// Or start it and poll yourself:
const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 50 });
const record = await xdive.waitForCrawl(crawlId, { timeoutMs: 600_000 });
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The seed URL. It is always fetched. |
| `max_depth` | integer | `2` | Link hops to follow from the seed. `0` fetches only the seed. |
| `max_pages` | integer | `100` | Pages to fetch, at most 500. `0` or a value above 500 means 500. |
| `same_domain` | boolean | `true` | Follow only links on the seed's registrable domain: for `https://docs.example.com`, links to `www.example.com` are followed, links to `example.org` are not. |
| `include_paths` | array of strings | none | RE2 patterns. When set, a discovered link is followed only if its URL path matches at least one. See [Path filters](#path-filters). |
| `exclude_paths` | array of strings | none | RE2 patterns. A discovered link whose path matches any of them is skipped. |
| `formats` | array | none | Representations for each page, as on [Scrape](https://xdive.dev/docs/scrape.md#formats). `screenshot` and `audio` are not available on crawls. Without `formats`, each page has a `markdown` field. |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip URLs that robots.txt disallows. They are reported by the [errors endpoint](#list-failed-pages). |
| `adaptive_fetch` | boolean | `true` | Plain HTTP first; on a block or bot filter, retry with a Chrome TLS fingerprint, and on a challenge page or JavaScript shell, render in a headless browser. `false` uses plain HTTP only. Cannot be combined with `impersonate` or `render`. |
| `impersonate` | boolean | `false` | Fetch every page with a Chrome TLS fingerprint and matching User-Agent. Turns `adaptive_fetch` off. |
| `render` | boolean | `false` | Render every page in a headless browser so JavaScript runs. Turns `adaptive_fetch` off. |
| `bypass_cache` | boolean | `false` | Neither read nor write the page cache. |
| `max_age` | integer | `0` | Cache freshness window in milliseconds, as on [Scrape](https://xdive.dev/docs/scrape.md#caching-and-max_age). `0` uses the server default. |
| `headers` | object | none | Extra request headers for every page fetch, with the same rules as [Scrape](https://xdive.dev/docs/scrape.md#custom-headers). |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses in every page with typed tokens. |
| `extract_schema` | object | none | Pull records from each page with CSS selectors. See [Structured records](#structured-records). |
| `llm_extract` | object | none | Pull records from each page with an LLM. Requires an LLM. |
| `webhook_url` | string | none | A public URL that receives a signed POST for the events in `webhook_events`. See [Webhooks](#webhooks). |
| `webhook_secret` | string | none | Key for the HMAC-SHA256 signature. Required unless the platform's signed-webhook policy is off. |
| `webhook_events` | array of strings | `["completed", "failed"]` | Any of `started`, `page`, `completed`, `failed`. |

### Structured records

`extract_schema` maps field names to CSS selectors. With `container`, each element it matches yields one record and the field selectors are evaluated inside it, which suits lists of products or posts; without it, each page yields one record. A field takes the element's text, or an attribute's value when `attr` is set. Invalid selectors are skipped.

```json
{
  "extract_schema": {
    "container": "article.post",
    "fields": [
      { "name": "title", "selector": "h2" },
      { "name": "link", "selector": "a", "attr": "href" }
    ]
  }
}
```

`llm_extract` describes fields in words instead: `fields` is a list of `name`, `description` and an optional `type` hint (`string`, `number` or `boolean`); `prompt` adds guidance and `multiple: true` asks for one record per item on list pages. It runs on the first 50 successfully fetched pages of a crawl. Both put their output in each page's `records`, a list of objects with string values.

## Response

The crawl is queued and the response is `202 Accepted`:

```json
{ "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d", "status": "queued" }
```

## Get a crawl

`GET https://xdive.dev/api/orgs/{id}/crawls/{crawlId}`

Returns the crawl's status, and its pages once it has finished. Batch scrapes use the same endpoint.

```json
{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "maxDepth": 2,
  "durationMs": 18342,
  "httpPages": 37,
  "cachePages": 4,
  "impersonatedPages": 1,
  "browserPages": 0,
  "fetchAttempts": 39,
  "fetchCostUnits": 40,
  "error": "",
  "createdAt": "2026-10-15T09:30:12.481557Z",
  "results": [
    {
      "url": "https://docs.example.com/guides/install",
      "title": "Install",
      "description": "Install the CLI on macOS, Linux and Windows.",
      "depth": 1,
      "status_code": 200,
      "word_count": 618,
      "fetched_at": "2026-10-15T09:30:16.020114Z",
      "fetch_route": "http",
      "fetch_attempts": 1,
      "fetch_cost_units": 1,
      "markdown": "# Install\n\nDownload the latest release..."
    }
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `id` | string | The crawl id. |
| `url` | string | The seed URL. |
| `status` | string | See [Status values](#status-values). |
| `pages` | integer | Pages in the result, including pages that failed. Set when the crawl finishes. |
| `maxDepth` | integer | The depth the crawl ran with. |
| `durationMs` | integer | Time the crawl took. |
| `httpPages`, `cachePages`, `impersonatedPages`, `browserPages` | integer | Pages by [fetch route](https://xdive.dev/docs/scrape.md#fetch-telemetry). |
| `fetchAttempts`, `fetchCostUnits` | integer | Sums of the pages' fetch attempts and cost units. |
| `error` | string | Why the crawl failed; empty otherwise. |
| `createdAt` | string | When the crawl was submitted. |
| `results` | array or null | The pages, in no particular order, once the crawl has finished; `null` before. |

Each page has the fields of the [scrape page object](https://xdive.dev/docs/scrape.md#the-page-object) except the `links` list, plus:

| Field | Type | Description |
|---|---|---|
| `depth` | integer | Link hops from the seed. |
| `markdown` | string | The page's Markdown, when the crawl had no `formats`. |
| `artifacts` | object | The requested formats, when the crawl had `formats`. |
| `records` | array | Output of `extract_schema` or `llm_extract`. |
| `error` | string | Why this page failed. |
| `error_code` | string | For sites that block data-centre networks: `PROXY_NOT_ENABLED` when your organization has no residential proxy access (or has used its monthly allowance), `SITE_RESTRICTED` when the proxy provider refused the destination. See [Government and banking sites](https://xdive.dev/docs/scrape.md#government-and-banking-sites). Otherwise absent. |
| `policy_refusal` | boolean | `true` when robots.txt disallowed the page. |

### Status values

| Status | Meaning |
|---|---|
| `queued` | Accepted and waiting for a worker. |
| `running` | A worker is fetching pages. |
| `success` | Finished, and at least one page was fetched without error. |
| `error` | Finished without a single successful page. `error` holds the first page's error, or `no pages crawled`. |
| `cancelled` | Cancelled with `DELETE`. `results` holds the pages fetched before the cancellation. |

`success`, `error` and `cancelled` are final.

### Polling with ETag and Retry-After

Every response carries an `ETag` that changes only when the crawl's status, page count or duration does. Send it back in `If-None-Match` and an unchanged crawl answers `304 Not Modified` with no body. While a crawl is `queued` or `running`, responses also carry `Retry-After: 2`, the suggested seconds between polls. The SDKs' `crawl`, `wait_for_crawl` (Python) and `waitForCrawl` (JavaScript) do both for you.

```bash
curl -i "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY" \
  -H 'If-None-Match: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"'
# HTTP/1.1 304 Not Modified
# Etag: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"
# Retry-After: 2
```

Polling, cancelling and listing errors are not counted against your plan's request rate limit.

## Cancel a crawl

`DELETE https://xdive.dev/api/orgs/{id}/crawls/{crawlId}`

Cancels a queued or running crawl or batch scrape and returns `{"id": "crl_…", "status": "cancelled"}`. A running crawl notices within about two seconds and saves the pages fetched so far; a crawl cancelled before a worker picked it up ends with no results and sends no webhook. Cancelling a finished crawl returns `409`.

SDK: `xdive.cancel_crawl(crawl_id)` in Python, `xdive.cancelCrawl(crawlId)` in JavaScript.

## List failed pages

`GET https://xdive.dev/api/orgs/{id}/crawls/{crawlId}/errors`

Returns only the pages that failed, so you can audit a large crawl without downloading every page. Pages refused by robots.txt are listed separately.

```json
{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "status": "success",
  "error": "",
  "errors": [
    { "url": "https://docs.example.com/guides/legacy", "error": "unexpected status 404" }
  ],
  "robots_blocked": ["https://docs.example.com/guides/internal"]
}
```

SDK: `xdive.crawl_errors(crawl_id)` in Python, `xdive.crawlErrors(crawlId)` in JavaScript.

## Path filters

`include_paths` and `exclude_paths` are [RE2](https://github.com/google/re2/wiki/Syntax) regular expressions matched against the path of each discovered link (`/guides/install`, without host or query). They are not anchored: `guides` matches anywhere in the path, `^/guides/` only at the start. An exclude match always wins over an include match. The filters decide which links are followed; the seed URL is fetched regardless. Invalid patterns are rejected with `400` before the crawl is queued.

## Webhooks

Set `webhook_url` to have the crawl POST its progress to you. The URL must resolve to a public address (`400` with `WEBHOOK_TARGET_DENIED` otherwise). By default the platform requires a `webhook_secret`; a request without one is rejected with `400` and `SIGNED_WEBHOOK_REQUIRED`.

### Events

| Subscription in `webhook_events` | Delivered event | When |
|---|---|---|
| `started` | `crawl.started` | A worker starts the crawl. |
| `page` | `crawl.page` | Each page is fetched, whether it succeeded or failed. |
| `completed` | `crawl.finished` | The crawl ends with status `success`. |
| `failed` | `crawl.finished` | The crawl ends with status `error` or `cancelled`. |

Without `webhook_events` you receive `completed` and `failed`. The delivered event name is in the `X-Crawler-Event` header and the payload's `event` field; for `crawl.finished`, read `status` to tell the outcomes apart.

### Payloads

```json
{
  "event": "crawl.finished",
  "crawlId": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "orgId": "org_5f2c9a1e7b3d4c6f8a0b2e4d6c8f1a3b",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "durationMs": 18342,
  "finishedAt": "2026-10-15T09:30:30.823910Z"
}
```

`crawl.finished` adds `error` when the crawl failed. `crawl.started` carries `event`, `crawlId`, `orgId`, `url` and `sentAt`. `crawl.page` adds a `page` object with the page's `url`, `title`, `error` and `depth`; fetch the content from the crawl endpoint.

### Verifying signatures

When the crawl has a `webhook_secret`, every delivery carries three headers:

| Header | Value |
|---|---|
| `X-Crawler-Timestamp` | Unix time in seconds when the delivery was signed. |
| `X-Crawler-Signature-V1` | `v1=` followed by the hex HMAC-SHA256 of the timestamp, a `.`, and the raw request body, keyed with your secret. |
| `X-Crawler-Signature` | The hex HMAC-SHA256 of the raw body alone. Kept for older receivers; prefer V1, which lets you reject replays. |

To verify, compute the HMAC over the exact bytes you received (before any JSON parsing), compare it with the header in constant time, and reject timestamps outside a window you choose, such as five minutes.

**Python**

```python
import hashlib
import hmac
import time

def verify_webhook(body: bytes, headers, secret: str, tolerance: int = 300) -> bool:
    timestamp = headers.get("X-Crawler-Timestamp", "")
    signature = headers.get("X-Crawler-Signature-V1", "").removeprefix("v1=")
    if not timestamp.isdigit() or abs(time.time() - int(timestamp)) > tolerance:
        return False
    expected = hmac.new(secret.encode(), timestamp.encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

**JavaScript**

```ts
import { createHmac, timingSafeEqual } from "node:crypto";

export function verifyWebhook(body: Buffer, headers: Record<string, string | undefined>, secret: string, toleranceSec = 300): boolean {
  const timestamp = headers["x-crawler-timestamp"] ?? "";
  const signature = (headers["x-crawler-signature-v1"] ?? "").replace(/^v1=/, "");
  if (!/^\d+$/.test(timestamp) || Math.abs(Date.now() / 1000 - Number(timestamp)) > toleranceSec) return false;
  const expected = createHmac("sha256", secret).update(`${timestamp}.`).update(body).digest("hex");
  return expected.length === signature.length && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}
```

### Delivery and retries

A delivery succeeds when your endpoint answers with a 2xx status within 10 seconds. `crawl.finished` is attempted up to three times, one and then two seconds apart; `crawl.started` and `crawl.page` are attempted once. Delivery is best effort, so treat the crawl endpoint as the source of truth.

## Limits

- At most 500 pages per crawl, and 2 link hops by default.
- A crawl runs for at most 2 minutes. Pages fetched by then are kept, and the crawl succeeds if any of them did.
- Each page is read up to 5 MB.
- `llm_extract` runs on at most 50 pages per crawl.
- At most 10 custom headers. The request body is limited to 1 MB.

Starting a crawl counts one crawl against your plan; the pages in the result, including pages that failed, are counted against your monthly page quota when it finishes. If the page quota is already used up, the crawl is refused with `402`. See [Usage and billing](https://xdive.dev/docs/usage.md).

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | Missing `url`, an invalid format or one not allowed on crawls, an invalid path pattern, header or `webhook_events` entry, a negative `max_age`, or `adaptive_fetch` combined with `impersonate` or `render`. |
| `400` | `WEBHOOK_TARGET_DENIED` | `webhook_url` is not a public URL. |
| `400` | `SIGNED_WEBHOOK_REQUIRED` | `webhook_url` was set without `webhook_secret`. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key lacks the `crawl` scope. |
| `404` | | No crawl with this id in your organization (status, cancel and errors). |
| `409` | | Cancelling a crawl that has already finished. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `501` | | `llm_extract` or an LLM format was requested and no LLM is configured. |
| `503` | | The crawl could not be queued. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) for a list of known URLs.
- [Map](https://xdive.dev/docs/map.md) to list a site's URLs without fetching content.
- [Scrape](https://xdive.dev/docs/scrape.md) for formats and fetch telemetry.

---

Source: https://xdive.dev/docs/crawl · All docs for agents: https://xdive.dev/llms.txt


---

# Batch scrape

Scrape a list of up to 100 known URLs as one asynchronous job. Each URL is fetched on its own, without following links, through the same pipeline as a crawl. The job has a crawl id, so you follow it, cancel it and list its failures with the [crawl endpoints](https://xdive.dev/docs/crawl.md#get-a-crawl), and it can notify a webhook.

`POST https://xdive.dev/api/orgs/{id}/batch/scrape`

Requires an API key with the `scrape` or the `crawl` scope. Submitting a batch counts against your plan's request rate limit.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/batch/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/pricing",
      "https://example.com/about",
      "https://example.com/blog/launch"
    ],
    "formats": ["markdown", "links"],
    "ignore_invalid_urls": true
  }'
# {"id":"crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f","status":"queued","urls":3,"invalid_urls":[]}
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
record = xdive.batch_scrape(
    [
        "https://example.com/pricing",
        "https://example.com/about",
        "https://example.com/blog/launch",
    ],
    formats=["markdown", "links"],
    ignore_invalid_urls=True,
)
for page in record["results"]:
    print(page["url"], page.get("error", ""))

# wait=False returns the job id instead of waiting:
job_id = xdive.batch_scrape(["https://example.com/pricing"], wait=False)
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const record = await xdive.batchScrape(
  [
    "https://example.com/pricing",
    "https://example.com/about",
    "https://example.com/blog/launch",
  ],
  { formats: ["markdown", "links"], ignore_invalid_urls: true },
);
for (const page of record.results ?? []) console.log(page.url, page.error ?? "");
```

The SDK methods submit the batch and poll it until it finishes (`wait=False` in Python returns the id instead). With cURL, poll `GET /orgs/{id}/crawls/{crawlId}` as described on [Crawl](https://xdive.dev/docs/crawl.md#polling-with-etag-and-retry-after).

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | required | 1 to 100 absolute `http` or `https` URLs. Blank entries are skipped and exact duplicates are scraped once. |
| `ignore_invalid_urls` | boolean | `false` | Skip invalid entries instead of rejecting the request. Skipped entries are returned in `invalid_urls`. |
| `formats` | array | none | Representations for each page, as on [Scrape](https://xdive.dev/docs/scrape.md#formats). `screenshot` and `audio` are not available. Without `formats`, each page has a `markdown` field. |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip URLs that robots.txt disallows. |
| `adaptive_fetch` | boolean | `true` | Plain HTTP first, escalating to a Chrome TLS fingerprint and then a headless browser for blocked pages, challenges and JavaScript shells. `false` uses plain HTTP only. |
| `bypass_cache` | boolean | `false` | Neither read nor write the page cache. |
| `max_age` | integer | `0` | Cache freshness window in milliseconds, as on [Scrape](https://xdive.dev/docs/scrape.md#caching-and-max_age). |
| `headers` | object | none | Extra request headers for every fetch, with the rules on [Scrape](https://xdive.dev/docs/scrape.md#custom-headers). |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses with typed tokens. |
| `webhook_url` | string | none | Receives signed event POSTs, as for [crawls](https://xdive.dev/docs/crawl.md#webhooks). |
| `webhook_secret` | string | none | HMAC-SHA256 signing key. Required unless the platform's signed-webhook policy is off. |
| `webhook_events` | array of strings | `["completed", "failed"]` | Any of `started`, `page`, `completed`, `failed`. |

A URL is valid when it parses as an absolute URL with an `http` or `https` scheme and a host.

## Response

The batch is queued and the response is `202 Accepted`:

```json
{
  "id": "crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f",
  "status": "queued",
  "urls": 3,
  "invalid_urls": []
}
```

| Field | Type | Description |
|---|---|---|
| `id` | string | The job id. Use it with the crawl endpoints. |
| `status` | string | `queued`. |
| `urls` | integer | Number of URLs that will be scraped, after removing blanks, duplicates and skipped invalid entries. |
| `invalid_urls` | array of strings | Entries skipped because they are not valid URLs. |

### Results

`GET /orgs/{id}/crawls/{crawlId}` returns the job in the [crawl format](https://xdive.dev/docs/crawl.md#get-a-crawl), with `url` set to `batch:` followed by the first URL and `maxDepth` set to `0`. Once the job is finished, `results` lists one page per URL in the order you submitted them; a URL that could not be fetched is a page with an `error`. The job's status is `success` when at least one URL succeeded.

To see only the failures, use [`GET /orgs/{id}/crawls/{crawlId}/errors`](https://xdive.dev/docs/crawl.md#list-failed-pages). To stop the job, [`DELETE /orgs/{id}/crawls/{crawlId}`](https://xdive.dev/docs/crawl.md#cancel-a-crawl); URLs not yet fetched are reported with the error `cancelled before fetch`.

## Limits and usage

- 100 URLs per batch, fetched four at a time.
- The job runs for at most 2 minutes; URLs not fetched by then are reported as failed pages.
- Each page is read up to 5 MB.

A batch counts as one crawl when submitted, and its pages, including failed ones, count against your monthly page quota when it finishes. If the quota is already used up, the batch is refused with `402`.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `urls` is empty or has more than 100 entries, no valid URL remains, a format is invalid or not allowed, or a header, `webhook_events` entry or `max_age` is invalid. |
| `400` | | Some entries are invalid and `ignore_invalid_urls` is not set. The body lists them in `invalid_urls`. |
| `400` | `WEBHOOK_TARGET_DENIED` | `webhook_url` is not a public URL. |
| `400` | `SIGNED_WEBHOOK_REQUIRED` | `webhook_url` was set without `webhook_secret`. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `scrape` nor the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `501` | | An LLM format (`json`, `product`, `menu`) was requested and no LLM is configured. |
| `503` | | The job could not be queued. |

```json
{
  "error": "urls contains invalid entries; pass ignore_invalid_urls to skip them",
  "invalid_urls": ["ftp://example.com/file", "example.com/no-scheme"]
}
```

## Related

- [Scrape](https://xdive.dev/docs/scrape.md) for one page, synchronously, with every option.
- [Crawl](https://xdive.dev/docs/crawl.md) for status, cancellation, errors and webhooks.
- [Extract](https://xdive.dev/docs/extract.md) to pull structured data across up to 10 URLs.

---

Source: https://xdive.dev/docs/batch-scrape · All docs for agents: https://xdive.dev/llms.txt


---

# Map

List a site's URLs quickly, without converting any content. Map reads the site's sitemaps and runs a links-only crawl, then returns the same-site URLs it found, optionally filtered by a search term. Use it to plan a crawl, pick pages for a [batch scrape](https://xdive.dev/docs/batch-scrape.md), or find where a topic lives on a site.

`POST https://xdive.dev/api/orgs/{id}/map`

Requires an API key with the `crawl` scope. Map is synchronous and returns within 45 seconds.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/map" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "search": "webhook", "limit": 100}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
urls = xdive.map("https://docs.example.com", search="webhook", limit=100)
print(len(urls), urls[:5])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const urls = await xdive.map("https://docs.example.com", { search: "webhook", limit: 100 });
console.log(urls.length, urls.slice(0, 5));
```

The SDK methods return the `urls` array.

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The page to start from. Its registrable domain is the site that is mapped. |
| `search` | string | none | Keep only URLs that contain this text, case-insensitively. Pages that do not match are still traversed, so matches deep in the site are found. |
| `limit` | integer | `5000` | Maximum URLs to return. Values above 5000, and `0`, mean 5000. |
| `max_depth` | integer | `2` | Link hops for the links-only crawl. |

## How URLs are found

1. **Sitemaps.** The `Sitemap` entries in the site's robots.txt are read, or `/sitemap.xml` when there are none. Sitemap indexes are followed.
2. **Links.** A links-only crawl starts at `url` and follows links up to `max_depth` hops, fetching at most 200 pages. Every same-site link on those pages is collected, including links to pages that were not fetched, so the result can be much larger than 200.

Only `http` and `https` URLs on the seed's registrable domain are returned (for `docs.example.com`, URLs on `www.example.com` count too). Duplicates are removed. Pages that robots.txt disallows are not fetched. Sitemap URLs come first, in sitemap order, followed by the crawl's discoveries.

Map is a discovery tool: it does not check that each URL still resolves, and it returns URLs in the form the sitemap or link used.

## Response

```json
{
  "count": 3,
  "urls": [
    "https://docs.example.com/guides/webhooks",
    "https://docs.example.com/reference/webhook-events",
    "https://docs.example.com/blog/signed-webhooks"
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `count` | integer | Number of URLs returned. |
| `urls` | array of strings | The URLs. |

## Usage

Map does not count pages against your quota, but it is refused with `402` once your monthly page quota is used up. It counts against your plan's request rate limit and concurrency limit like other synchronous endpoints.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `url` is missing or not an absolute `http` or `https` URL, or the body is not valid JSON. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key lacks the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and rate limits.

## Related

- [Crawl](https://xdive.dev/docs/crawl.md) to fetch and convert the pages.
- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) to scrape a chosen subset.
- [llms.txt](https://xdive.dev/docs/llms-txt.md) to build an index of a site for language models.

---

Source: https://xdive.dev/docs/map · All docs for agents: https://xdive.dev/llms.txt


---

# Search

Search the web, news or images and get ranked results. Optionally read the top results and get the passages that answer your query, scrape them in any format, or have a short cited answer written from them. Use Search when you want results to choose from; use [Research](https://xdive.dev/docs/research.md) when you want a finished answer with verified citations.

`POST https://xdive.dev/api/orgs/{id}/search`

Requires an API key with the `search` scope. New keys get `crawl` and `scrape` by default, so add `search` when you create the key. Search is synchronous: the search and any page reads share a 25-second budget, and `include_answer` gets 30 more seconds to write.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/search" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "query": "rotate API keys without downtime",
    "limit": 5,
    "freshness": "py",
    "search_depth": "advanced"
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
response = xdive.search(
    "rotate API keys without downtime",
    limit=5,
    freshness="py",
    search_depth="advanced",
)
for result in response["results"]:
    best = result.get("passages", [{}])[0].get("text", "")
    print(result["rank"], result["url"], best[:120])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const response = await xdive.search("rotate API keys without downtime", {
  limit: 5,
  freshness: "py",
  search_depth: "advanced",
});
for (const result of response.results) {
  console.log(result.rank, result.url, result.passages?.[0]?.text.slice(0, 120));
}
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | required | The search query. At most 400 characters and 50 words. |
| `sources` | array of strings | `["web"]` | Any of `web`, `news`, `images`. Results from several sources are interleaved. |
| `limit` | integer | `10` | Results to return, 1 to 50. |
| `country` | string | none | Two-letter country code to localize results, such as `US` or `DE`. |
| `search_language` | string | none | Language code such as `en` or `en-us`. |
| `freshness` | string | none | Limit results by publication date. See [Freshness](#freshness). |
| `safe_search` | string | `moderate` | `off`, `moderate` or `strict`. |
| `include_domains` | array of strings | none | Only return results from these hosts. At most 20. See [Domain filters](#domain-filters). |
| `exclude_domains` | array of strings | none | Never return results from these hosts. At most 20. |
| `search_depth` | string | `basic` | `basic` returns the engine's results. `advanced` also reads the top results and attaches query-relevant `passages` and a `relevance_score`. See [Search depth](#search-depth). |
| `auto_parameters` | boolean | `false` | Infer `sources` and `freshness` from the wording of the query when you did not set them. |
| `scrape` | object | none | Scrape the top results. See [Scraping results](#scraping-results). |
| `include_answer` | boolean | `false` | Also write a short answer with citations from the top results. Needs an LLM. See [Answer](#answer). |

### Freshness

| Value | Results published |
|---|---|
| `pd` | In the past day. |
| `pw` | In the past week. |
| `pm` | In the past month. |
| `py` | In the past year. |
| `2026-01-01to2026-06-30` | Between two dates, inclusive, written `YYYY-MM-DDtoYYYY-MM-DD`. |

### Domain filters

Entries are host names such as `example.com`, without a scheme, path or port. A filter matches the host and its subdomains: `example.com` matches `docs.example.com` but not `notexample.com`. Filters are applied after the engine ranks results, so a narrow `include_domains` can return fewer than `limit` results.

### Search depth

With `search_depth: "advanced"`, the top 8 web and news results are read in parallel, through the page cache, within 6 seconds. Each result that could be read gains:

- `page`: the page's `url`, `title` and main-content `markdown`, so you need no second fetch;
- `passages`: up to 3 excerpts most relevant to the query, best first;
- `relevance_score`: the score of the best passage.

Results keep the search engine's order. Advanced search does not re-rank them by `relevance_score`; sort by it yourself if that suits your use. Results that could not be read in time have no `passages`. Each page read counts as one page against your quota.

Passage scores are BM25 relevance scores of the passage against the query: higher is more relevant, and `0` means no query term in common, so such passages are left out.

### Auto parameters

`auto_parameters: true` fills in parameters you left unset from the query's wording. Words such as "news", "latest", "breaking", "today" or "announced" add `news` to `sources`. "today", "yesterday", "breaking", "right now", "this week" or "latest" set `freshness` to `pw`; "this month", "recent" or "recently" set `pm`; the current year written in the query sets `py`. The parameters used are returned in `parameters`.

### Scraping results

`scrape` fetches the top web and news results through the scrape pipeline and attaches each one as `page`, with `passages` and `relevance_score` computed from its content.

| Field | Type | Default | Description |
|---|---|---|---|
| `limit` | integer | `3` | Results to scrape, 0 to 5. `0` means the default. |
| `formats` | array of strings | `["markdown"]` | Format names, as on [Scrape](https://xdive.dev/docs/scrape.md#formats). Format options are not accepted here. |
| `respect_robots` | boolean | `true` | Skip pages that robots.txt disallows. |
| `main_content_only` | boolean | `true` | Main-content extraction. |

A result that could not be scraped has `scrape_error` instead of `page`. With `advanced` and Markdown-only scraping, pages already read are reused; with other formats they are fetched again, but a page is never counted twice.

### Answer

`include_answer: true` writes an answer of at most about 150 words from the top results, citing them as `[1]`, `[2]`. Citation numbers are checked against the sources in code, so the answer cannot cite a page that was not read. If you do not pass `scrape`, the top 5 results are scraped for it.

The answer needs an LLM configured for your organization. Problems with the answer never fail the search: `answer` then holds an `error` instead.

## Response

```json
{
  "query": "rotate API keys without downtime",
  "provider": "brave",
  "search_depth": "advanced",
  "results": [
    {
      "id": "4f9c2a7e1b3d5f80",
      "source": "web",
      "rank": 1,
      "title": "Rotating API keys safely",
      "url": "https://docs.example.com/security/key-rotation",
      "description": "Issue a second key, deploy it, then revoke the first.",
      "published_at": "2026-03-02T00:00:00Z",
      "page": {
        "url": "https://docs.example.com/security/key-rotation",
        "title": "Rotating API keys safely",
        "depth": 0,
        "markdown": "# Rotating API keys safely\n\nKeep two keys active during a rotation..."
      },
      "passages": [
        {
          "text": "Keep two keys active during a rotation. Issue the new key, roll it out to every client, confirm traffic has moved, then revoke the old key.",
          "score": 6.218,
          "start": 412
        }
      ],
      "relevance_score": 6.218
    },
    {
      "id": "b07e19d2c4a6f835",
      "source": "web",
      "rank": 2,
      "title": "Zero-downtime credential rotation",
      "url": "https://blog.example.org/zero-downtime-rotation",
      "description": "A checklist for rotating secrets in production."
    }
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `query` | string | The query as searched. |
| `provider` | string | The search engine used, such as `brave` or `serper`; several engines are joined with `+` (`brave+serper`) when their results are fused. |
| `search_depth` | string | The `search_depth` you sent, empty when omitted. |
| `parameters` | object | With `auto_parameters`: the `sources` and `freshness` that were used. |
| `results` | array | The results, best first. |
| `answer` | object | With `include_answer`: `text`, `citations` and `confidence`, or `error`. |

### Result fields

| Field | Type | Description |
|---|---|---|
| `id` | string | Stable identifier of the result. |
| `source` | string | `web`, `news` or `images`. |
| `rank` | integer | Position in this response, from 1. |
| `title`, `url`, `description` | string | As returned by the engine. |
| `published_at` | string | Publication time, when the engine knows it. |
| `image` | object | For image results: `url`, `thumbnail`, `width`, `height`. |
| `page` | object | The page's content, from `advanced` or `scrape`. With `scrape`, a full [scrape page object](https://xdive.dev/docs/scrape.md#the-page-object). |
| `passages` | array | Up to 3 query-relevant excerpts, best first. Each has `text`, `score` and `start`, the byte offset of the excerpt in the page's text. |
| `relevance_score` | number | The best passage's score. |
| `scrape_error` | string | Why `scrape` could not fetch this result. |

### Answer fields

| Field | Type | Description |
|---|---|---|
| `text` | string | The answer, with citation markers such as `[1]`. |
| `citations` | array | The sources the answer cites: `number`, `url`, `title` and a supporting `quote`. |
| `confidence` | string | `high`, `medium`, `low` or `none`, from how many sources the answer cites. See [Research](https://xdive.dev/docs/research.md#confidence). |
| `error` | string | Present instead of the fields above when no answer could be written, for example when no LLM is configured. |

## Usage

Each request counts as one search. Pages read by `advanced` and pages fetched by `scrape` count as pages. Identical engine queries are cached for an hour (10 minutes for news or `freshness` searches) and are answered faster, but still count as a search. A request is refused with `402` when your search quota, or for `advanced` and `scrape` your page quota, is used up.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | Missing or too long `query`, an unknown source, `limit` outside 1 to 50, an invalid `country`, `search_language`, `freshness` or `safe_search`, more than 20 domain filters or a filter that is not a host name, an invalid `search_depth`, `scrape.limit` outside 0 to 5, or an unknown format in `scrape.formats`. |
| `402` | | The monthly search quota, or the page quota for `advanced` and `scrape`, is used up. |
| `403` | | The key lacks the `search` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `502` | | The search engine failed. |
| `503` | | No search engine is configured on the server. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Research](https://xdive.dev/docs/research.md) for an answer with verified citations from a full search-and-read run.
- [Scrape](https://xdive.dev/docs/scrape.md) for formats and their options.
- [Usage and billing](https://xdive.dev/docs/usage.md) for what is metered.

---

Source: https://xdive.dev/docs/search · All docs for agents: https://xdive.dev/llms.txt


---

# Research

Ask a question and get an answer written from the web, in which every claim cites a source that was actually read. A research run searches, reads the best pages, keeps the passages relevant to the question, and has a language model write the answer; the citations are then checked in code. Use it for questions you would otherwise answer by searching and reading several pages yourself. For raw results to choose from, use [Search](https://xdive.dev/docs/search.md).

`POST https://xdive.dev/api/orgs/{id}/research`

Requires an API key with the `search` scope, an LLM configured for your organization and a search engine on the server. Research is synchronous: a quick run takes seconds and is stopped after 45 seconds, a deep run after 150 seconds.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/research" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "query": "How long does Example Cloud keep audit logs by default?",
    "depth": "quick",
    "freshness": "py"
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.research(
    "How long does Example Cloud keep audit logs by default?",
    depth="quick",
    freshness="py",
)
print(result["answer"])
for citation in result["citations"]:
    print(f"[{citation['number']}] {citation['url']}")
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.research("How long does Example Cloud keep audit logs by default?", {
  depth: "quick",
  freshness: "py",
});
console.log(result.answer);
for (const citation of result.citations) console.log(`[${citation.number}] ${citation.url}`);
```

> **Tip:**
>
> The SDKs time out requests after 120 seconds by default. For `depth: "deep"`, raise it: `XDive(timeout=180)` in Python, `new XDive({ timeoutMs: 180_000 })` in JavaScript.

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | required | The question, at most 500 characters. Answers are written in the language of the question. |
| `depth` | string | `quick` | `quick`: one search round, seconds. `deep`: planned sub-queries and up to three rounds. See [How a run works](#how-a-run-works). |
| `output` | string | `answer` for quick, `report` for deep | `answer`: a direct answer of at most 150 words. `report`: a structured Markdown report of about 400 to 800 words. |
| `schema` | object | none | A JSON Schema. The result is returned as `data` instead of prose. See [Structured output](#structured-output). |
| `max_sources` | integer | `6` quick, `15` deep | Sources to keep for the answer. At most 10 above the default (16 quick, 25 deep); larger values are capped. |
| `include_domains` | array of strings | none | Only use results from these hosts and their subdomains, as on [Search](https://xdive.dev/docs/search.md#domain-filters). |
| `exclude_domains` | array of strings | none | Never use results from these hosts. |
| `freshness` | string | none | `pd`, `pw`, `pm`, `py` or a `YYYY-MM-DDtoYYYY-MM-DD` range, as on [Search](https://xdive.dev/docs/search.md#freshness). |
| `country` | string | none | Two-letter country code to localize the searches. |
| `no_cache` | boolean | `false` | Run fresh instead of serving a cached result, and do not cache this one. See [Caching](#caching). |

## How a run works

### Quick

1. The question is searched (10 results). Pages found by several queries rank higher, and no site gets more than two places.
2. Up to 10 candidate pages are read in parallel, through the page cache, within 6 seconds. Once 3 pages have arrived, the run waits at most 0.7 seconds for more instead of holding the answer for the slowest site. Pages that were not read in time contribute their search snippet instead; such sources are marked `snippet: true` and rank below pages that were read.
3. Each page is split into passages and the passages most relevant to the question are kept, up to 4 per source, across up to 6 sources.
4. The model writes the answer from those numbered passages, citing them as `[1]`, `[2]`.

### Two-step questions

Some questions describe a thing indirectly and then ask about it, such as "Name a co-founder of the company that develops the V8 JavaScript engine." A single search for that question finds pages about V8, not about the company's founders.

In quick mode, while the first search runs, the model triages the question. When it finds a two-step question, the intermediate thing (here, the company) is identified from the pages already read, never from the model's own knowledge; if those pages do not name it, one search for it is made. The remaining question is then searched with the name filled in, adding up to 4 sources. `usage.plan` reports the triage (`single`, `multi_hop`, or `unavailable` when it failed or was not ready 2 seconds after the first round, and was ignored) and `usage.hops` is `1` when a bridge was made. The triage is skipped when you pass a `schema`.

### Follow-up search

When the writer reports that the sources lack a fact the question needs, it names a search for exactly that fact. Quick mode makes that one follow-up search, reads up to 4 more sources within 4 seconds, and rewrites the answer. If the writer asked for more without naming a search, a short gap check decides what to search. The follow-up is skipped when a two-step bridge was already made or a `schema` was passed, and if the rewrite fails the first answer is returned. A quick run makes at most three search rounds in total.

If the writer still returns no answer, it is asked once more to answer from the sources it has, or to say in one sentence that they do not contain the answer and what they do say, with citations. A run that found sources does not come back with an empty answer.

### Deep

The model first plans 2 to 4 sub-queries that cover the question's aspects; they are searched together with the question. After each round a gap check lists up to 3 queries for what is still missing, and the next round searches them, for up to 3 rounds. Each round reads up to 24 candidates within 20 seconds, and up to 15 sources are kept. The default output is a report.

## Citations

Sources are numbered from 1 and passed to the model as untrusted data, with instructions to ignore any instructions inside them; the model has no tools, so a page can change the words of the answer but not what is fetched or cited. After the model writes, every `[n]` marker is checked in code against the numbered sources:

- a number that matches no source is removed from the text;
- groups such as `[1, 3]` and ranges such as `[1-3]` are expanded, and repeats collapse;
- `citations` lists exactly the sources the answer cites, in number order, each with a `quote`: up to 300 characters of that source's most relevant passage.

A fabricated citation cannot survive this check. `sources` lists every source that was given to the model, cited or not.

### Confidence

`confidence` summarizes how well the answer is grounded:

| Value | When |
|---|---|
| `high` | The answer cites at least 2 sources, and at least half of all sources. |
| `medium` | The answer cites at least 1 source. |
| `low` | The answer cites no source. |
| `none` | There is no answer. |

## Structured output

Pass `schema` to get JSON instead of prose. The model fills the schema from the sources and uses `null` for anything they do not support. The response has `data` instead of `answer`, and because structured output carries no citation markers, every source is listed in `citations`. The schema must be a JSON object.

```json
{
  "query": "Audit log retention and export options in Example Cloud",
  "schema": {
    "type": "object",
    "properties": {
      "default_retention_days": { "type": "integer" },
      "max_retention_days": { "type": "integer" },
      "export_formats": { "type": "array", "items": { "type": "string" } }
    }
  }
}
```

## Caching

An identical request from your organization within 10 minutes is answered from a cache, with `cached: true`, and is not metered. Requests count as identical after trimming the query and filling in the default `depth` and `output`. Identical requests that arrive while a run is in progress share that run. Failed runs are not cached. Set `no_cache: true` to force a fresh run.

## Response

```json
{
  "query": "How long does Example Cloud keep audit logs by default?",
  "answer": "Example Cloud keeps audit log entries for 90 days by default [1]. Enterprise plans can extend retention to 365 days [1][2].",
  "citations": [
    {
      "number": 1,
      "url": "https://docs.example.com/security/audit-logs",
      "title": "Audit logs",
      "quote": "Audit log entries are retained for 90 days. On the Enterprise plan, retention can be extended to 365 days from the security settings."
    },
    {
      "number": 2,
      "url": "https://example.com/pricing",
      "title": "Pricing",
      "quote": "Enterprise: SSO, SCIM, audit log retention up to 365 days, and a 99.95% uptime SLA."
    }
  ],
  "sources": [
    {
      "number": 1,
      "url": "https://docs.example.com/security/audit-logs",
      "title": "Audit logs",
      "passages": [
        {
          "text": "Audit log entries are retained for 90 days. On the Enterprise plan, retention can be extended to 365 days from the security settings.",
          "score": 7.412,
          "start": 318
        }
      ],
      "score": 7.412
    },
    {
      "number": 2,
      "url": "https://example.com/pricing",
      "title": "Pricing",
      "passages": [
        {
          "text": "Enterprise: SSO, SCIM, audit log retention up to 365 days, and a 99.95% uptime SLA.",
          "score": 4.106,
          "start": 2210
        }
      ],
      "score": 4.106
    },
    {
      "number": 3,
      "url": "https://blog.example.org/audit-logging-compared",
      "title": "Audit logging compared",
      "passages": [
        {
          "text": "Example Cloud stores audit events for three months unless you upgrade.",
          "score": 3.877,
          "start": 0
        }
      ],
      "score": 2.326,
      "snippet": true
    }
  ],
  "queries": ["How long does Example Cloud keep audit logs by default?"],
  "confidence": "high",
  "usage": {
    "searches": 1,
    "pages_read": 5,
    "rounds": 1,
    "plan": "single",
    "duration_ms": 4210,
    "search_ms": 812,
    "read_ms": 1630,
    "model_ms": 1544
  }
}
```

| Field | Type | Description |
|---|---|---|
| `query` | string | The question, trimmed. |
| `answer` | string | The answer, with citation markers. Absent when `schema` was passed. |
| `data` | object | The filled schema, when `schema` was passed. |
| `citations` | array | The sources the answer cites: `number`, `url`, `title`, `quote`. |
| `sources` | array | Every source given to the model. See below. |
| `queries` | array of strings | Every search query the run made, in order, including sub-queries, bridge and follow-up searches. |
| `confidence` | string | `high`, `medium`, `low` or `none`. See [Confidence](#confidence). |
| `usage` | object | The work the run did. See below. |
| `cached` | boolean | `true` when the result came from the cache. Absent otherwise. |

### Sources

| Field | Type | Description |
|---|---|---|
| `number` | integer | The number the answer cites. |
| `url`, `title` | string | The page. |
| `passages` | array | Up to 4 relevant excerpts, best first: `text`, `score` (BM25 relevance) and `start` (byte offset in the page's text). |
| `score` | number | The best passage's score, lowered for snippet sources. Sources are ordered by it. |
| `snippet` | boolean | `true` when the page could not be read in time and the passage is the search engine's snippet. Absent otherwise. |

### Usage

| Field | Type | Description |
|---|---|---|
| `searches` | integer | Search engine queries that completed. |
| `pages_read` | integer | Pages fetched and read. |
| `rounds` | integer | Search-and-read rounds. |
| `hops` | integer | Two-step bridges made (quick mode). Absent when 0. |
| `plan` | string | The two-step triage: `single`, `multi_hop` or `unavailable`. Quick mode without `schema` only. |
| `duration_ms` | integer | Total run time. |
| `search_ms`, `read_ms`, `model_ms` | integer | Time spent searching, reading pages and waiting for the model, summed across rounds. |

A fresh run counts `searches` against your search quota and `pages_read` against your page quota. Cached results cost nothing. A request is refused with `402` when either quota is already used up.

## Errors

Errors with a `code` also have `"success": false`.

| Status | Code | When |
|---|---|---|
| `400` | | Missing `query`, a query over 500 characters, an invalid `depth` or `output`, or a body that is not valid JSON. |
| `402` | | The monthly search or page quota is used up. |
| `403` | | The key lacks the `search` scope. |
| `404` | `NO_SOURCES` | No readable source was found for the question. Broaden the question or loosen domain and freshness filters. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `501` | | No LLM is configured for the organization, or no search engine on the server. |
| `502` | | The model or another upstream service failed. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |
| `504` | `RESEARCH_TIMEOUT` | The run did not finish within 45 seconds (quick) or 150 seconds (deep). Try `depth: "quick"` or a narrower question. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Search](https://xdive.dev/docs/search.md) for ranked results with passages, and `include_answer` for a short answer from the top results.
- [Extract](https://xdive.dev/docs/extract.md) to fill a schema from pages you already know.
- [Usage and billing](https://xdive.dev/docs/usage.md) for what is metered.

---

Source: https://xdive.dev/docs/research · All docs for agents: https://xdive.dev/llms.txt


---

# Extract

Pull structured data out of up to 10 pages in one call. The pages are scraped concurrently, their content is combined, and your LLM answers a prompt or fills a JSON Schema from all of them together. Use Extract when the data you need is spread across a few known pages, such as a product's pricing, docs and changelog; for one page, the [`json` format on Scrape](https://xdive.dev/docs/scrape.md#formats) does the same per page.

`POST https://xdive.dev/api/orgs/{id}/extract`

Requires an API key with the `scrape` or the `crawl` scope, and an LLM configured for your organization. Extract is synchronous and stops after 90 seconds.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/extract" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://example.com/docs/limits"],
    "prompt": "List each plan with its monthly price and API rate limit.",
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthly_price_usd": { "type": "number" },
              "requests_per_minute": { "type": "integer" }
            }
          }
        }
      }
    },
    "show_sources": true
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.extract(
    ["https://example.com/pricing", "https://example.com/docs/limits"],
    prompt="List each plan with its monthly price and API rate limit.",
    schema={
        "type": "object",
        "properties": {
            "plans": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "monthly_price_usd": {"type": "number"},
                        "requests_per_minute": {"type": "integer"},
                    },
                },
            }
        },
    },
    show_sources=True,
)
print(result["data"]["plans"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.extract(["https://example.com/pricing", "https://example.com/docs/limits"], {
  prompt: "List each plan with its monthly price and API rate limit.",
  schema: {
    type: "object",
    properties: {
      plans: {
        type: "array",
        items: {
          type: "object",
          properties: {
            name: { type: "string" },
            monthly_price_usd: { type: "number" },
            requests_per_minute: { type: "integer" },
          },
        },
      },
    },
  },
  show_sources: true,
});
console.log(result.data);
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | required | 1 to 10 absolute `http` or `https` URLs. |
| `prompt` | string | none | What to extract. At least one of `prompt` and `schema` is required. |
| `schema` | object | none | A JSON Schema for the result. Without a prompt, the model is asked to extract the data the schema describes. |
| `main_content_only` | boolean | `true` | Main-content extraction on each page: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip pages that robots.txt disallows. Such pages count as failed sources. |
| `show_sources` | boolean | `false` | Include the per-URL `sources` list in a successful response. |

## How it works

1. Each URL is scraped as Markdown through the standard pipeline (page cache, robots.txt, the same fetch escalation as [Scrape](https://xdive.dev/docs/scrape.md#fetch-telemetry)), three at a time, with 25 seconds per page.
2. Pages that fail are skipped. The others are combined in the order of `urls`, each labelled with its number and URL and cut to its first 24,000 bytes, so one long page cannot crowd out the rest.
3. The model fills `schema` or answers `prompt` from the combined text and returns JSON.

The request succeeds as long as one page was scraped. Use `show_sources` to see which pages failed and why.

## Response

```json
{
  "success": true,
  "data": {
    "plans": [
      { "name": "Starter", "monthly_price_usd": 0, "requests_per_minute": 60 },
      { "name": "Team", "monthly_price_usd": 49, "requests_per_minute": 600 }
    ]
  },
  "sources_scraped": 2,
  "sources": [
    { "url": "https://example.com/pricing", "title": "Pricing" },
    { "url": "https://example.com/docs/limits", "title": "Rate limits" }
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true`. |
| `data` | object | The JSON the model produced, following `schema` when you gave one. |
| `sources_scraped` | integer | Pages that were scraped and used. |
| `sources` | array | With `show_sources`: one entry per URL with `url`, and `title` on success or `error` on failure. |

## Usage

Each page scraped counts as one page against your monthly page quota; pages that failed do not. The request is refused with `402` when the quota is already used up. The model call runs on your organization's LLM.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `urls` is empty, has more than 10 entries or contains an invalid URL, or neither `prompt` nor `schema` was given. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `scrape` nor the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `501` | | No LLM is configured for the organization. |
| `502` | | No page could be scraped, or the model call failed (`extraction failed: …`). |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

When no page could be scraped, the body lists every URL and its error:

```json
{
  "success": false,
  "error": "no source could be scraped",
  "sources": [
    { "url": "https://example.com/pricing", "error": "disallowed by robots.txt" },
    { "url": "https://example.com/docs/limits", "error": "unexpected status 404" }
  ]
}
```

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Scrape](https://xdive.dev/docs/scrape.md) with the `json`, `product` or `menu` format for one page.
- [Crawl](https://xdive.dev/docs/crawl.md) with `llm_extract` for records from every page of a site.
- [Research](https://xdive.dev/docs/research.md) with `schema` to fill a schema from pages found by searching.

---

Source: https://xdive.dev/docs/extract · All docs for agents: https://xdive.dev/llms.txt


---

# llms.txt

Generate an [llms.txt](https://llmstxt.org) file for a site: a Markdown index of its most important pages, with titles and descriptions, that language models and agents can read to find their way around. Optionally get `llms-full.txt` too, with the full Markdown of those pages in one file.

`POST https://xdive.dev/api/orgs/{id}/llmstxt`

Requires an API key with the `crawl` or the `scrape` scope. The request is synchronous and stops after 2 minutes.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/llmstxt" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "max_urls": 20, "full_text": true}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.llms_txt("https://docs.example.com", max_urls=20, full_text=True)

with open("llms.txt", "w") as f:
    f.write(result["llms_txt"])
with open("llms-full.txt", "w") as f:
    f.write(result["llms_full_txt"])
```

**JavaScript**

```ts
import { writeFile } from "node:fs/promises";
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true });

await writeFile("llms.txt", result.llms_txt as string);
await writeFile("llms-full.txt", result.llms_full_txt as string);
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The site's start page, an absolute `http` or `https` URL. It becomes the first entry, and its title and description head the file. |
| `max_urls` | integer | `10` | Pages to include, the start page among them. At most 20; larger values are capped. |
| `full_text` | boolean | `false` | Also return `llms_full_txt` with each page's full Markdown. |

## How pages are chosen

1. The site is mapped the way [Map](https://xdive.dev/docs/map.md) does it: its sitemaps plus a links-only crawl of up to 50 pages, 2 hops deep, within 30 seconds.
2. The start page comes first. The other URLs are ordered by path depth, shallowest first, so section landing pages come before deep articles. URLs ending in `.xml` or `.json`, and paths containing `login` or `wp-admin`, are skipped.
3. The chosen pages are scraped as main-content Markdown, four at a time, through the page cache and with robots.txt respected. Pages that fail are left out of the files.

## Response

```json
{
  "success": true,
  "llms_txt": "# Example Docs\n\n> Guides and API reference for Example.\n\n## Pages\n\n- [Example Docs](https://docs.example.com): Guides and API reference for Example.\n- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.\n- [Authentication](https://docs.example.com/authentication): API keys and scopes.\n",
  "pages": 3,
  "llms_full_txt": "# Example Docs\nSource: https://docs.example.com\n\n...\n\n---\n\n# Quickstart\nSource: https://docs.example.com/quickstart\n\n...\n\n---\n\n"
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true`. |
| `llms_txt` | string | The index file. |
| `pages` | integer | Pages scraped and included. |
| `llms_full_txt` | string | With `full_text`: every included page's Markdown, at most 500,000 bytes. |

The `llms_txt` file has this shape:

```markdown
# Example Docs

> Guides and API reference for Example.

## Pages

- [Example Docs](https://docs.example.com): Guides and API reference for Example.
- [Quickstart](https://docs.example.com/quickstart): Make your first request in five minutes.
- [Authentication](https://docs.example.com/authentication): API keys and scopes.
```

The heading is the start page's title (the host name when it has none or could not be scraped), followed by its meta description as a quote. Each entry links a page by its title, followed by its meta description when it has one. In `llms_full_txt`, each page starts with its title as a heading and a `Source:` line, and pages are separated by `---`. Pages are added until the file reaches 500,000 bytes, and the file is cut at that size.

## Usage

Each page scraped counts as one page against your monthly page quota. The request is refused with `402` when the quota is already used up.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `url` is missing or not an absolute `http` or `https` URL. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `crawl` nor the `scrape` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `502` | | No page could be scraped. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Map](https://xdive.dev/docs/map.md) for the full list of a site's URLs.
- [Crawl](https://xdive.dev/docs/crawl.md) to convert more than 20 pages.
- [MCP](https://xdive.dev/docs/mcp.md) to give an assistant live access to sites instead of a static file.

---

Source: https://xdive.dev/docs/llms-txt · All docs for agents: https://xdive.dev/llms.txt


---

# MCP (Claude, ChatGPT, Cursor)

XDive runs a hosted [Model Context Protocol](https://modelcontextprotocol.io) server, so AI assistants and agents can search the web, read pages, crawl sites, extract data and research questions with cited sources. Tools call the same API as your own code, so scopes, quotas, caching and the audit log apply exactly as they do to a direct request. Most clients connect with OAuth: you paste one URL, sign in to XDive and approve; no API key is copied anywhere.

`POST https://xdive.dev/api/mcp`

The server URL is `https://xdive.dev/api/mcp` (self-hosted: `https://your-host/api/mcp`). Use it exactly as written, without a trailing slash.

## Overview

The server exposes eight tools, always in this order. Every tool only reads the public web; none changes anything.

| Tool | What it does | Returns |
|---|---|---|
| `search` | Searches the web, news or images; can also read the top results and write a short cited answer. | JSON: `results`, each with `id`, `title`, `url`, `text` and `passages`, and `answer` when asked. `id` is the result's URL, ready for `fetch`. |
| `fetch` | Reads one URL as Markdown. | JSON: `id`, `title`, `text`, `url` and `metadata`; `metadata.next_offset` is set when the page continues. |
| `research` | Searches, reads the best sources and writes an answer in which every claim cites a source. | Text: the answer or a Markdown report, the numbered sources, and a confidence line. |
| `scrape` | Reads one URL in any scrape format. | Markdown text by default; JSON for other formats. |
| `map` | Lists a site's URLs from its sitemaps and links without reading them. | Text: the number of URLs, then one URL per line. |
| `crawl` | Crawls a site from a seed URL. | Text: a summary and an excerpt of each page, or the crawl's id while it runs. |
| `crawl_status` | Shows a crawl or batch scrape by id. | The same view as `crawl`. |
| `extract` | Pulls structured data from up to 10 URLs with a prompt and/or a JSON Schema. | JSON: the [Extract](https://xdive.dev/docs/extract.md) response. |

### Tool arguments

| Tool | Arguments |
|---|---|
| `search` | `query` (required); `sources`: `web`, `news`, `images` (default `web`); `limit`: 1 to 50 (default 10); `scrape_top`: 0 to 5; `include_answer`; `country` (ISO code such as `us`); `freshness`: `pd`, `pw`, `pm`, `py` or a date range |
| `fetch` | `id` (required): the URL; `offset`; `max_chars` |
| `research` | `query` (required); `depth`: `quick` (default) or `deep`; `output`: `answer` or `report`; `max_sources`: 1 to 30; `freshness`; `country`; `include_domains` |
| `scrape` | `url` (required); `formats`: `markdown` (default), `text`, `html`, `rawHtml`, `links`, `images`, `summary`, `branding`, `product`, `menu`; `redact_pii`; `max_age` (milliseconds); `offset`; `max_chars` |
| `map` | `url` (required); `search`; `limit`: 1 to 5,000 |
| `crawl` | `url` (required); `max_depth`: 0 to 10 (default 2); `max_pages`: 1 to 1,000 (default 100); `include_paths` and `exclude_paths` (RE2 patterns); `wait` (default `true`); `max_chars` |
| `crawl_status` | `id` (required): the `crl_…` id; `max_chars` |
| `extract` | `urls` (required): 1 to 10 URLs; `prompt`; `schema` (JSON Schema) |

The server also sends instructions that tell the model which tool fits which need (search then fetch for precise reading, research for a synthesized and cited answer, map before a crawl). Clients that support server instructions add them to the model's context.

Each tool call is metered like the API call behind it: `search` and `research` like [Search](https://xdive.dev/docs/search.md) and [Research](https://xdive.dev/docs/research.md), `fetch` and `scrape` like [Scrape](https://xdive.dev/docs/scrape.md), and so on. `research`, `extract` and `search` with `include_answer` use the language model configured for your organization. See [Usage and billing](https://xdive.dev/docs/usage.md).

## Connect a client

Pick the tab that matches how you want to authenticate. **OAuth** is the default for every client below: the client discovers XDive's authorization server, opens a browser, and you sign in, choose an organization and click **Allow**. **API key** is for headless agents and clients where you prefer a key: create a dedicated key with the `scrape`, `crawl` and `search` scopes under **API Keys**, keep it in an environment variable, and send it as `Authorization: Bearer ck_live_…` (or `X-API-Key`). Never put a key in the URL.

### ChatGPT

Custom MCP servers are added from **Plugins** in ChatGPT on the web; you don't need to turn on developer mode first.

1. Open **Plugins**, select **+**, then **Add custom MCP server**.
2. Name it XDive and paste the server URL under **Connection**.
3. Choose OAuth for authentication, review the risk warning, and select **I understand and want to continue**.
4. Select **Create as a plugin**, sign in to XDive, choose an organization and click **Allow**. ChatGPT then lists the eight tools.
5. In a chat, type `@` and select XDive.

```text
https://xdive.dev/api/mcp
```

Custom MCP servers are available on ChatGPT Business and Enterprise/Edu workspaces (workspace admins control who may add them) and are not available on mobile. Pro accounts can connect servers for read and fetch actions, which covers all of XDive's tools. Deep research can use the server too: `search` and `fetch` return the result and document shapes it expects.

### Claude (claude.ai and Claude Desktop)

Claude connects to the server from Anthropic's cloud, for claude.ai, Claude Desktop and the mobile apps alike, so a self-hosted server must be reachable from the internet.

1. Open **Customize → Connectors**, select **+ Add**, then **Add custom connector**.
2. Name it XDive, paste the server URL, and select **Continue**. Review the detected authentication and continue.
3. Under **OAuth client**, keep the default. XDive accepts Claude's published client identity (a client ID metadata document) and also supports automatic registration.
4. Select **Add**, then **Connect**. Sign in to XDive, choose an organization and click **Allow**.
5. Turn the connector on for a conversation from **+ → Connectors**.

On Team and Enterprise plans an owner adds the connector once under **Organization settings → Connectors → Add → Custom → Web**, and each member then connects it with their own XDive account from **Customize → Connectors**. Free plans allow one custom connector.

The connector form also has **Request headers** for an API key (`Authorization: Bearer ck_live_…`). On Team and Enterprise plans everyone who uses the connector then shares that key, so prefer OAuth.

If you sign in to Claude Code with the same claude.ai account, connectors you added on claude.ai are available there as well.

### Claude Code

**OAuth**

```bash
claude mcp add --transport http --scope user xdive https://xdive.dev/api/mcp
# then, in a Claude Code session: /mcp, select xdive, and sign in in the browser
# or from your shell:
claude mcp login xdive
```

**API key**

```bash
claude mcp add --transport http --scope user xdive https://xdive.dev/api/mcp \
  --header "Authorization: Bearer $XDIVE_API_KEY"
```

`--scope user` makes the server available in every project; leave it out to add it to the current project only. The shell expands `$XDIVE_API_KEY` when you run the command, so the key is stored in your Claude Code configuration. To share a project configuration without the key, commit a `.mcp.json` that reads it from the environment:

```json
{
  "mcpServers": {
    "xdive": {
      "type": "http",
      "url": "https://xdive.dev/api/mcp",
      "headers": { "Authorization": "Bearer ${XDIVE_API_KEY}" }
    }
  }
}
```

`claude -p` and Agent SDK runs cannot open a browser: sign in once in an interactive session (or with `claude mcp login xdive`) first. Claude Code warns when a tool result exceeds 10,000 tokens and stops at 25,000 by default (`MAX_MCP_OUTPUT_TOKENS`); XDive's default page window of 24,000 characters stays well below that.

### Codex

The Codex CLI, the Codex IDE extension and the ChatGPT desktop app share one configuration file, `~/.codex/config.toml` (or `.codex/config.toml` in a trusted project).

**OAuth**

```toml
[mcp_servers.xdive]
url = "https://xdive.dev/api/mcp"
tool_timeout_sec = 120
```

**API key**

```toml
[mcp_servers.xdive]
url = "https://xdive.dev/api/mcp"
bearer_token_env_var = "XDIVE_API_KEY"
tool_timeout_sec = 120
```

With OAuth, sign in once from the CLI; the IDE extension and the desktop app also show an **Authenticate** button next to the server under **MCP servers**:

```bash
codex mcp login xdive
```

You can add the server from the UI instead: in the ChatGPT desktop app open **Settings → MCP servers → Add server** (in the IDE extension, the gear menu → **MCP servers**), choose **Streamable HTTP**, enter the URL, save and restart.

> **Tip: Set tool_timeout_sec**
>
> Codex stops waiting for a tool after 60 seconds by default. XDive answers every call within about 55 seconds (research and crawl return a "still running" result after about 40 seconds rather than fail), so `tool_timeout_sec = 120` leaves room for slow networks. All tools are marked read-only, so `default_tools_approval_mode = "writes"` lets Codex run them without asking each time.

**Codex cloud tasks.** Cloud environments are configured in ChatGPT (repositories, setup scripts, environment variables, internet access) and do not read your local `config.toml`; OpenAI's documentation says local MCP servers may not be available in the cloud and documents no way to attach a custom MCP server to a cloud environment. The documented route to MCP tools across ChatGPT surfaces is a plugin; see [Plugins](#plugins).

### Cursor

Add the server to `~/.cursor/mcp.json` for every project, or to `.cursor/mcp.json` in one project.

**OAuth**

```json
{
  "mcpServers": {
    "xdive": {
      "url": "https://xdive.dev/api/mcp"
    }
  }
}
```

**API key**

```json
{
  "mcpServers": {
    "xdive": {
      "url": "https://xdive.dev/api/mcp",
      "headers": { "Authorization": "Bearer ${env:XDIVE_API_KEY}" }
    }
  }
}
```

With OAuth, Cursor registers itself and asks you to sign in when it first connects; manage the server from **Customize**. Cursor resolves `${env:NAME}` in `url` and `headers`; remote servers do not support `envFile`, so set the variable in your shell profile.

### VS Code (GitHub Copilot)

Run **MCP: Add Server** from the Command Palette, choose **HTTP**, and enter the URL, or write the configuration yourself:

- `.mcp.json` at the workspace root (`mcpServers`): shared with your team and compatible tools.
- `~/.copilot/mcp-config.json` (`mcpServers`): your own servers, in every workspace.
- `.vscode/mcp.json` (`servers`): for VS Code features such as input variables.

**OAuth**

```json
{
  "mcpServers": {
    "xdive": {
      "type": "http",
      "url": "https://xdive.dev/api/mcp"
    }
  }
}
```

**API key**

```json
{
  "inputs": [
    { "type": "promptString", "id": "xdive-api-key", "description": "XDive API key", "password": true }
  ],
  "servers": {
    "xdive": {
      "type": "http",
      "url": "https://xdive.dev/api/mcp",
      "headers": { "Authorization": "Bearer ${input:xdive-api-key}" }
    }
  }
}
```

The OAuth example uses the `mcpServers` format of `.mcp.json` and `~/.copilot/mcp-config.json`; in `.vscode/mcp.json` the key is `servers`. With OAuth, VS Code registers itself with XDive and opens the browser when the server first starts. The API key variant goes in `.vscode/mcp.json`: VS Code asks for the key once and stores it securely. Copilot harness sessions do not receive servers that need an `${input:…}` value.

### Gemini CLI

**OAuth**

```bash
gemini mcp add --transport http --scope user xdive https://xdive.dev/api/mcp
# then, inside Gemini CLI, run: /mcp auth xdive
```

**API key**

```json
{
  "mcpServers": {
    "xdive": {
      "httpUrl": "https://xdive.dev/api/mcp",
      "headers": { "Authorization": "Bearer $XDIVE_API_KEY" }
    }
  }
}
```

`gemini mcp add` writes to `~/.gemini/settings.json` with `--scope user`, or to the project's `.gemini/settings.json` by default. When editing the file yourself, use `httpUrl`: in Gemini CLI, `url` means the older SSE transport. String values in `settings.json` can reference environment variables as `$NAME` or `${NAME}`.

### Windsurf (Devin Desktop)

Windsurf is now Devin Desktop. Its agents read MCP servers from `~/.config/devin/mcp_config.json` (`%APPDATA%\devin\mcp_config.json` on Windows).

**OAuth**

```json
{
  "mcpServers": {
    "xdive": {
      "url": "https://xdive.dev/api/mcp"
    }
  }
}
```

**API key**

```json
{
  "mcpServers": {
    "xdive": {
      "url": "https://xdive.dev/api/mcp",
      "headers": { "Authorization": "Bearer ${env:XDIVE_API_KEY}" }
    }
  }
}
```

With OAuth you are asked to sign in when the server is first used. With the Devin CLI, `devin mcp add -s user xdive https://xdive.dev/api/mcp` writes the same entry and `devin mcp login xdive` signs in.

### Clients that only run local servers

A client that can only start local (stdio) servers can reach XDive through [mcp-remote](https://github.com/geelen/mcp-remote), which runs locally and forwards to the remote server. It handles OAuth itself: the first run opens a browser to sign in.

**OAuth**

```json
{
  "mcpServers": {
    "xdive": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://xdive.dev/api/mcp"]
    }
  }
}
```

**API key**

```json
{
  "mcpServers": {
    "xdive": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://xdive.dev/api/mcp", "--header", "Authorization:${XDIVE_AUTH}"],
      "env": { "XDIVE_AUTH": "Bearer ck_live_…" }
    }
  }
}
```

mcp-remote substitutes `${XDIVE_AUTH}` from `env`. Writing the header without a space after the colon avoids a bug in some clients that do not quote arguments containing spaces.

Self-hosted deployments can also build the repository's stdio server, `crawler-mcp` (`cmd/mcp`), which calls the API with `XDIVE_API_KEY`, `XDIVE_ORG_ID` and, optionally, `XDIVE_API_URL` (default `https://xdive.dev/api`).

Client menus change often. If a step above no longer matches, see the vendor's guide: [ChatGPT](https://help.openai.com/en/articles/12584461-developer-mode-apps-and-full-mcp-connectors-in-chatgpt-beta), [Claude](https://support.claude.com/en/articles/11175166-getting-started-with-custom-connectors-using-remote-mcp), [Claude Code](https://code.claude.com/docs/en/mcp), [Codex](https://learn.chatgpt.com/docs/extend/mcp?surface=cli), [Cursor](https://cursor.com/docs/context/mcp), [VS Code](https://code.visualstudio.com/docs/copilot/customization/mcp-servers), [Gemini CLI](https://github.com/google-gemini/gemini-cli/blob/main/docs/tools/mcp-server.md), [Devin Desktop](https://docs.devin.ai/desktop/cascade/mcp).

## Use it from code

Model APIs that call remote MCP servers on your behalf can use XDive directly. They connect from the provider's infrastructure, so the server must be publicly reachable, and they take a bearer token: pass an API key.

### OpenAI Responses API

**cURL**

```bash
curl https://api.openai.com/v1/responses \
  -H "Authorization: Bearer $OPENAI_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6-astra",
    "tools": [{
      "type": "mcp",
      "server_label": "xdive",
      "server_description": "Search the web, read pages and research questions with cited sources.",
      "server_url": "https://xdive.dev/api/mcp",
      "authorization": "'"$XDIVE_API_KEY"'",
      "require_approval": "never"
    }],
    "input": "How does HTTP/3 differ from HTTP/2? Cite your sources."
  }'
```

**Python**

```python
import os
from openai import OpenAI

client = OpenAI()
response = client.responses.create(
    model="gpt-6-astra",
    tools=[{
        "type": "mcp",
        "server_label": "xdive",
        "server_description": "Search the web, read pages and research questions with cited sources.",
        "server_url": "https://xdive.dev/api/mcp",
        "authorization": os.environ["XDIVE_API_KEY"],
        "require_approval": "never",
    }],
    input="How does HTTP/3 differ from HTTP/2? Cite your sources.",
)
print(response.output_text)
```

**JavaScript**

```ts
import OpenAI from "openai";

const client = new OpenAI();
const response = await client.responses.create({
  model: "gpt-6-astra",
  tools: [{
    type: "mcp",
    server_label: "xdive",
    server_description: "Search the web, read pages and research questions with cited sources.",
    server_url: "https://xdive.dev/api/mcp",
    authorization: process.env.XDIVE_API_KEY,
    require_approval: "never",
  }],
  input: "How does HTTP/3 differ from HTTP/2? Cite your sources.",
});
console.log(response.output_text);
```

`authorization` is sent to XDive as a bearer token. `require_approval: "never"` skips OpenAI's per-call approval; remove it to review each call first. `allowed_tools` (for example `["search", "fetch", "research"]`) limits which tools the model sees. In the OpenAI Agents SDK, `HostedMCPTool(tool_config={…})` takes the same object.

### Anthropic Messages API

The MCP connector is a beta feature: send the `anthropic-beta: mcp-client-2025-11-20` header (the SDKs take `betas`). Each server in `mcp_servers` needs a matching `mcp_toolset` entry in `tools`.

**cURL**

```bash
curl https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "anthropic-beta: mcp-client-2025-11-20" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-opus-5-5",
    "max_tokens": 2000,
    "messages": [{"role": "user", "content": "How does HTTP/3 differ from HTTP/2? Cite your sources."}],
    "mcp_servers": [{
      "type": "url",
      "url": "https://xdive.dev/api/mcp",
      "name": "xdive",
      "authorization_token": "'"$XDIVE_API_KEY"'"
    }],
    "tools": [{"type": "mcp_toolset", "mcp_server_name": "xdive"}]
  }'
```

**Python**

```python
import os
import anthropic

client = anthropic.Anthropic()
response = client.beta.messages.create(
    model="claude-opus-5-5",
    max_tokens=2000,
    messages=[{"role": "user", "content": "How does HTTP/3 differ from HTTP/2? Cite your sources."}],
    mcp_servers=[{
        "type": "url",
        "url": "https://xdive.dev/api/mcp",
        "name": "xdive",
        "authorization_token": os.environ["XDIVE_API_KEY"],
    }],
    tools=[{"type": "mcp_toolset", "mcp_server_name": "xdive"}],
    betas=["mcp-client-2025-11-20"],
)
print(response.content)
```

**JavaScript**

```ts
import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();
const response = await anthropic.beta.messages.create({
  model: "claude-opus-5-5",
  max_tokens: 2000,
  messages: [{ role: "user", content: "How does HTTP/3 differ from HTTP/2? Cite your sources." }],
  mcp_servers: [{
    type: "url",
    url: "https://xdive.dev/api/mcp",
    name: "xdive",
    authorization_token: process.env.XDIVE_API_KEY,
  }],
  tools: [{ type: "mcp_toolset", mcp_server_name: "xdive" }],
  betas: ["mcp-client-2025-11-20"],
});
console.log(response.content);
```

### Raw JSON-RPC

Any HTTP client can talk to the server. With the current protocol revision (`2026-07-28`) there is no handshake: each request carries its protocol version, client information and capabilities in `params._meta`, and repeats the version and method in headers. Start with `server/discover`:

```bash
curl https://xdive.dev/api/mcp \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -H "Mcp-Method: server/discover" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "server/discover",
    "params": {
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {"name": "curl", "version": "1.0"},
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  }'
```

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "_meta": {
      "io.modelcontextprotocol/serverInfo": {
        "name": "xdive",
        "title": "XDive",
        "version": "1.6.0",
        "websiteUrl": "https://xdive.dev",
        "icons": [{ "src": "https://xdive.dev/icon-512.png", "mimeType": "image/png", "sizes": ["512x512"] }, { "src": "https://xdive.dev/icon-192.png", "mimeType": "image/png", "sizes": ["192x192"] }]
      }
    },
    "cacheScope": "private",
    "capabilities": { "tools": {} },
    "instructions": "Web access tools. Pick by need: …",
    "resultType": "complete",
    "supportedVersions": ["2026-07-28", "2025-11-25", "2025-06-18", "2025-03-26", "2024-11-05"],
    "ttlMs": 600000
  }
}
```

Call a tool with `tools/call`. A tool call also needs the `Mcp-Name` header, set to the tool's name:

```bash
curl https://xdive.dev/api/mcp \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -H "MCP-Protocol-Version: 2026-07-28" \
  -H "Mcp-Method: tools/call" \
  -H "Mcp-Name: fetch" \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "fetch",
      "arguments": {"id": "https://example.com"},
      "_meta": {
        "io.modelcontextprotocol/protocolVersion": "2026-07-28",
        "io.modelcontextprotocol/clientInfo": {"name": "curl", "version": "1.0"},
        "io.modelcontextprotocol/clientCapabilities": {}
      }
    }
  }'
```

```json
{
  "jsonrpc": "2.0",
  "id": 2,
  "result": {
    "_meta": { "io.modelcontextprotocol/serverInfo": { "name": "xdive", "title": "XDive", "version": "1.6.0" } },
    "content": [
      {
        "text": "{\"id\":\"https://example.com\",\"metadata\":{\"offset\":0,\"total_chars\":182},\"text\":\"# Example Domain\\n\\nThis domain is for use in documentation examples…\",\"title\":\"Example Domain\",\"url\":\"https://example.com\"}",
        "type": "text"
      }
    ],
    "isError": false,
    "resultType": "complete"
  }
}
```

The tool's output is the `text` of the single content block (`serverInfo` is shortened here; it is the same object `server/discover` returns). Clients on older revisions open with `initialize` instead; see [Protocol support](#protocol-support).

## Plugins

A plugin package for Codex (ChatGPT desktop app and Codex CLI) and Claude Code bundles this server, signed in with OAuth, with a `web-research` skill that teaches the agent when to use each tool, how to read long pages in parts, how to continue a long research run and how to cite sources.

The package is ready but not yet published: it is not listed in the OpenAI plugin directory, Anthropic's directory or a public marketplace. Until it is, connect the server directly as described above; the plugin adds the skill on top of the same tools. Once it is listed in the OpenAI plugin directory, which ChatGPT and Codex share, installing it there is the documented way to bring the tools to ChatGPT surfaces that do not read `config.toml`.

## Authentication

### OAuth

Clients that support MCP authorization sign in without any configuration beyond the URL:

1. An unauthenticated request gets `401` with `WWW-Authenticate: Bearer resource_metadata="https://xdive.dev/.well-known/oauth-protected-resource/api/mcp"`.
2. That protected-resource metadata (RFC 9728) names the authorization server, whose metadata (RFC 8414) is at `/.well-known/oauth-authorization-server`.
3. The client identifies itself in one of two ways. Clients that publish a **client ID metadata document** use its `https` URL as their `client_id`; XDive fetches and validates the document (at most 5 KB, no redirects, `client_id` must equal the URL, public clients only) when you approve, so nothing is registered in advance. Other clients register at `/api/oauth/register` (dynamic client registration, RFC 7591). Either way clients are public (no secret); redirect URIs must be `https`, or `http` on a loopback address, where any port is accepted (RFC 8252).
4. You sign in on XDive's consent screen at `/oauth/authorize`, which names the app and where it will send you, choose the organization the app may use, and click **Allow** or **Deny**.
5. The client exchanges the code at `/api/oauth/token`. PKCE with `S256` is required; the only scope is `mcp`.

Access tokens last 1 hour and refresh tokens 30 days; refresh tokens rotate on every use, and reusing a spent code or refresh token revokes the whole grant. Tokens are bound to `/api/mcp`: the REST API rejects them. A grant can scrape, crawl and search for the organization you chose, and stops working when you leave the organization, the organization is suspended, or your account loses access.

Members see every app connected to an organization under **API Keys → Connected apps** and can revoke one there; approvals and revocations are recorded in the audit log. A revoked app has to ask you again.

### API keys

Send an organization API key as `Authorization: Bearer ck_live_…` or `X-API-Key: ck_live_…`. The key decides the organization, so no organization id is needed. Each tool needs one scope:

| Scope | Tools |
|---|---|
| `search` | `search`, `research` |
| `scrape` | `fetch`, `scrape` |
| `crawl` | `map`, `crawl`, `crawl_status` |
| `scrape` or `crawl` | `extract` |

New keys get `scrape` and `crawl` by default; add `search` when you create a key for MCP. Key expiry and IP allowlists apply as usual. See [Authentication](https://xdive.dev/docs/authentication.md).

## How results are shaped

Every tool returns one text block, sized to fit client limits on tool output.

- **Page windows.** `fetch` and `scrape` return up to 24,000 characters of a page (about 6,000 tokens). Set `max_chars` between 1,000 and 60,000 to change the window. When the page continues, `fetch` sets `metadata.next_offset` and `scrape` ends with `[Page continues: … Call scrape again with offset=N to read on.]`; call again with that `offset`. Later parts are read from a copy cached for up to 10 minutes, so all parts come from the same fetch.
- **Search.** Each result has `id` (its URL), `title`, `url`, `text` (the search engine's description) and, when available, `passages` that match the query. With `scrape_top`, each read result adds up to 4,000 characters of `content`. `answer` is present with `include_answer`.
- **Research.** The answer cites sources as `[1]`, `[2]`; a `Sources:` list follows, then a line such as `Confidence: high · 6 pages read · 12.4s`. A research call waits about 40 seconds. A run that takes longer keeps going on the server, and the tool answers that it is still running and asks the model to call again with exactly the same arguments in about 15 seconds (30 for `deep`). The repeated call returns the finished answer, which is kept for 10 minutes, instead of starting over.
- **Crawl.** `crawl` waits about 40 seconds. A finished crawl shows a summary and an excerpt of each page, sharing the output budget (at least 1,500 characters per page) and listing pages that did not fit; read any of them with `fetch`. A crawl still running returns its id for `crawl_status`. Set `wait: false` to get the id at once.
- **Map** lists URLs until the output limit and then says how many more there are; narrow it with `search` or `limit`.
- **Size limit.** No tool result exceeds 60,000 characters; anything longer is cut and marked `…[truncated]`.
- **Errors** come back as tool results with `isError: true` and a sentence such as `Error 403 (SITE_RESTRICTED): …` or `Error 402: …`, so the model can react. See [Errors and limits](https://xdive.dev/docs/errors.md).
- **Annotations.** Every tool is marked `readOnlyHint: true`, `destructiveHint: false` and `openWorldHint: true`; all except `crawl` are also `idempotentHint: true`. Clients use these hints to decide when to ask before running a tool. Tools carry titles and XDive's icon.

## Protocol support

- **Transport:** Streamable HTTP. Requests are `POST` with a JSON-RPC body of at most 4 MiB, and responses are plain JSON (`Cache-Control: no-store`); the server never opens an SSE stream.
- **Stateless:** there are no sessions, and any server instance can answer any request. `GET` and `DELETE` on the endpoint return `405` with `Allow: POST`.
- **Revisions:** `2026-07-28`, plus `2025-11-25`, `2025-06-18`, `2025-03-26` and `2024-11-05` through `initialize`. The era is chosen per request: a request whose `params._meta` names `io.modelcontextprotocol/protocolVersion` is served under the current revision; anything else follows the `initialize` handshake, where the server echoes a version it supports or offers `2025-11-25`.
- **Current revision (`2026-07-28`):** the `MCP-Protocol-Version` header must equal the version in `_meta`, `Mcp-Method` must equal the method, and `Mcp-Name` must equal the tool name on `tools/call` (`=?base64?…?=` encoding is accepted). A mismatch returns `400` with error `-32020`; an unsupported version returns `400` with error `-32022` and the list of supported versions; an unknown method returns `404`. Results carry `resultType: "complete"` and the server's identity in `_meta`; `server/discover` and `tools/list` add `ttlMs` (600,000) and `cacheScope: "private"`.
- **Earlier revisions:** `MCP-Protocol-Version`, when sent, must name a supported version. JSON-RPC batches (a `2025-03-26` feature) are answered as an array. Notifications get `202` with no body.
- **Capabilities:** tools only. The tool list does not change at runtime.
- **Timing:** each request is answered within about 55 seconds.

## Troubleshooting

| Symptom | Cause and fix |
|---|---|
| `401` when connecting, or the client asks you to sign in again | The token expired, was revoked under **Connected apps**, or was issued for another server. Sign in again from the client (for example `/mcp` in Claude Code, `codex mcp login xdive`, `/mcp auth xdive` in Gemini CLI). With an API key, `invalid API key` means the key is wrong, expired, revoked or used from an address outside its allowlist. |
| A tool returns `Error 403` about a scope | The API key lacks the tool's scope. Create a key with `scrape`, `crawl` and `search`. |
| `403` with `ACCOUNT_PENDING_APPROVAL` or `ACCOUNT_REJECTED` | The XDive account is waiting for an administrator's approval, or was not approved. The same codes appear when the client exchanges or refreshes a token. |
| `403 organization is suspended` | Contact the organization's owner or the administrator. |
| The consent screen says `unknown client` or `client metadata: …` | For a registered client, connect again so it re-registers (unused registrations are removed after a day). For a metadata-document client, the message names what is wrong with its document. |
| A tool call times out | Raise the client's tool timeout: `tool_timeout_sec = 120` in Codex. If research answers that it is still running, call it again with the same arguments. |
| `405 Method Not Allowed` | The client sent `GET`, for example because it is configured for the SSE transport. Configure Streamable HTTP (`"type": "http"` or `"streamable-http"`, `httpUrl` in Gemini CLI). |
| `404` or failed discovery | Check the URL: `https://xdive.dev/api/mcp`, with `/api` and no trailing slash. |
| The client only starts local servers | Use [mcp-remote](#clients-that-only-run-local-servers). |
| Self-hosted: sign-in fails with `resource does not identify this MCP server` | `APP_URL` must be the public `https` origin clients use; tokens are bound to `APP_URL` followed by `/api/mcp`. See [Self-hosting](https://xdive.dev/docs/self-hosting.md). |

---

Source: https://xdive.dev/docs/mcp · All docs for agents: https://xdive.dev/llms.txt


---

# LangChain

Both SDKs ship LangChain integrations: three agent tools (search, scrape and research) for LangChain in Python and LangChain.js, and a document loader for Python. The tools return compact text built for a model's context window: the search tool returns the passages of each result that match the query, not whole pages.

## Install

**Python**

```bash
pip install "xdive[langchain]"   # installs langchain-core
```

**JavaScript**

```bash
npm install xdive @langchain/core zod
```

In Python the tools live in `xdive.langchain`; in JavaScript they are imported from `xdive/langchain`. `@langchain/core` and `zod` are optional peer dependencies of the npm package, needed only for this entry point.

The tools read `XDIVE_API_KEY` (and `XDIVE_BASE_URL`) like the SDK clients. The key needs the `search` scope for the search and research tools and the `scrape` scope for the scrape tool; the loader's crawl mode needs `crawl`.

## Tools

| Class | Tool name | Input | Returns |
|---|---|---|---|
| `XDiveSearchTool` | `xdive_search` | `query`, `limit` (default 5, 1 to 20), `freshness` (`pd`, `pw`, `pm` or `py`, optional) | JSON array of results, each with its relevant passages. |
| `XDiveScrapeTool` | `xdive_scrape` | `url` | The page's Markdown, cut at 60,000 characters. |
| `XDiveResearchTool` | `xdive_research` | `query`, `depth` (`quick` or `deep`, default `quick`) | JSON object with a cited answer and its citations. |

The tool names and descriptions are what the model sees when it decides which tool to call. The search tool's description points the model to `xdive_research` when it needs a synthesized answer.

### xdive_search

Calls [search](https://xdive.dev/docs/search.md) with `search_depth: "advanced"` and `auto_parameters: true`: the API reads the top web results and keeps the excerpts most relevant to the query, and infers news sources and a time range from the wording when the query asks for recent information. The tool returns a JSON array:

```json
[
  {
    "title": "What's new in Python packaging",
    "url": "https://packaging.example.org/news/license-metadata",
    "snippet": "License-Expression and License-File fields replace the old classifiers…",
    "passages": [
      "PEP 639 adds the License-Expression field, which holds an SPDX expression such as MIT OR Apache-2.0…",
      "Build backends copy the files named by license-files into the .dist-info/licenses directory…"
    ],
    "relevance": 11.82
  }
]
```

| Field | Description |
|---|---|
| `title`, `url` | The search result. |
| `snippet` | The search engine's description of the result. |
| `passages` | Up to 3 excerpts of the page that best match the query, best first. Empty for results the API did not read or found nothing relevant in. |
| `relevance` | The score of the best passage: BM25 relevance to the query, not normalized; higher is more relevant. Absent (JavaScript) or `null` (Python) when `passages` is empty. |

The API reads at most 8 results per call, keeping the search engine's order. A call counts as one search plus one page for each result read.

In Python, `limit` is clamped to 1 to 20; in JavaScript the input schema rejects values outside that range and `freshness` must be one of the four values.

### xdive_scrape

Calls [scrape](https://xdive.dev/docs/scrape.md) with the Markdown format and returns the Markdown as plain text, cut at 60,000 characters so one long page cannot fill the context. In JavaScript the input schema requires a valid URL. A call counts as one page.

### xdive_research

Calls [research](https://xdive.dev/docs/research.md) and returns a JSON object:

```json
{
  "answer": "PEP 639 replaces license classifiers with an SPDX License-Expression field [1] and records license files in the wheel [2].",
  "confidence": "high",
  "citations": [
    { "n": 1, "url": "https://peps.python.org/pep-0639/", "title": "PEP 639" },
    { "n": 2, "url": "https://packaging.example.org/news/license-metadata", "title": "What's new in Python packaging" }
  ]
}
```

`answer` holds the answer text with numbered citation markers; `n` is the marker's number. `quick` research takes seconds, `deep` one to two minutes. A call counts the searches and pages the research used; a result served from the short-lived cache is not counted. See [Usage and billing](https://xdive.dev/docs/usage.md).

### Sharing a client

Each tool creates its own client from the environment unless you give it one. To share configuration and the cached organization id, pass a client:

**Python**

```python
from xdive import XDive
from xdive.langchain import XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool

xdive = XDive(api_key="ck_live_…", base_url="https://your-host/api")
tools = [
    XDiveSearchTool(client=xdive),
    XDiveScrapeTool(client=xdive),
    XDiveResearchTool(client=xdive),
]
```

**JavaScript**

```ts
import { XDive } from "xdive";
import { XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool } from "xdive/langchain";

const xdive = new XDive({ apiKey: process.env.XDIVE_API_KEY, baseUrl: "https://your-host/api" });
const tools = [new XDiveSearchTool(xdive), new XDiveScrapeTool(xdive), new XDiveResearchTool(xdive)];
```

In Python the client is the `client` keyword argument; in JavaScript it is the constructor's only argument.

## Build an agent

The tools work with any LangChain chat model that supports tool calling. This example uses LangChain's `create_agent` / `createAgent` with an Anthropic model, which needs `langchain` and the Anthropic integration installed and `ANTHROPIC_API_KEY` set:

**Python**

```bash
pip install "xdive[langchain]" langchain langchain-anthropic
```

**JavaScript**

```bash
npm install xdive @langchain/core zod langchain @langchain/anthropic
```

**Python**

```python
from langchain.agents import create_agent

from xdive import XDive
from xdive.langchain import XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool

xdive = XDive()  # reads XDIVE_API_KEY
agent = create_agent(
    model="anthropic:claude-opus-5-5",
    tools=[
        XDiveSearchTool(client=xdive),
        XDiveScrapeTool(client=xdive),
        XDiveResearchTool(client=xdive),
    ],
    system_prompt="Answer from the web and cite the URLs you used.",
)

result = agent.invoke(
    {"messages": [{"role": "user", "content": "What changed in the latest Node.js LTS release?"}]}
)
print(result["messages"][-1].content)
```

**JavaScript**

```ts
import { createAgent } from "langchain";
import { XDive } from "xdive";
import { XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool } from "xdive/langchain";

const xdive = new XDive(); // reads XDIVE_API_KEY
const agent = createAgent({
  model: "anthropic:claude-opus-5-5",
  tools: [new XDiveSearchTool(xdive), new XDiveScrapeTool(xdive), new XDiveResearchTool(xdive)],
  systemPrompt: "Answer from the web and cite the URLs you used.",
});

const result = await agent.invoke({
  messages: [{ role: "user", content: "What changed in the latest Node.js LTS release?" }],
});
console.log(result.messages.at(-1)?.content);
```

A typical run calls `xdive_search` to find sources, reads the passages, and calls `xdive_scrape` on a result when it needs the whole page. For questions that need several sources, `xdive_research` returns a cited answer in one call.

The tools raise the SDK's errors (`XDiveError` and its subclasses in Python, the matching error classes in JavaScript) after the client's own retries. How the agent reacts to a failed tool call depends on your LangChain version's tool-error handling. See [Python SDK errors](https://xdive.dev/docs/sdk-python.md#errors) and [JavaScript SDK errors](https://xdive.dev/docs/sdk-javascript.md#errors).

## Document loader

`XDiveLoader` (Python) loads one page or a whole site as LangChain `Document`s, for indexing in a vector store or passing to a chain.

```python
from xdive.langchain import XDiveLoader

page = XDiveLoader("https://example.com/pricing").load()
site = XDiveLoader("https://docs.example.com", mode="crawl", max_pages=100, max_depth=3).load()

for doc in site:
    print(doc.metadata["source"], len(doc.page_content))
```

`XDiveLoader(url, *, mode="scrape", client=None, **options)`

| Argument | Description |
|---|---|
| `url` | The page to scrape, or the seed URL of the crawl. |
| `mode` | `"scrape"` loads the one URL; `"crawl"` crawls from it and loads every page. Any other value raises `ValueError`. |
| `client` | An `XDive` client to use. Defaults to `XDive()`, which reads the environment. |
| `**options` | Passed to `XDive.scrape` or `XDive.crawl`: crawl parameters such as `max_pages`, `max_depth`, `include_paths` and `exclude_paths`, and the client's `timeout` and `poll_interval` for crawls. |

`load()` returns a list; `lazy_load()` yields the documents one by one. Each document's `page_content` is the page's Markdown. Its `metadata` holds `source` (the page URL) and, when the page has them, `url`, `title`, `description`, `lang` and `fetched_at`. In crawl mode, pages that failed are skipped.

A crawl waits for the whole crawl to finish before the first document is returned; the client's default wait is 600 seconds (`timeout`), after which `CrawlTimeout` is raised.

The JavaScript package has no loader class. Build documents from a crawl with `pageMarkdown`:

```ts
import { Document } from "@langchain/core/documents";
import { XDive, pageMarkdown } from "xdive";

const xdive = new XDive();
const record = await xdive.crawl("https://docs.example.com", { max_pages: 100 });
const docs = (record.results ?? [])
  .filter((page) => !page.error)
  .map((page) => new Document({ pageContent: pageMarkdown(page), metadata: { source: page.url, title: page.title } }));
```

## Related

- [LlamaIndex](https://xdive.dev/docs/llamaindex.md)
- [Python SDK](https://xdive.dev/docs/sdk-python.md) and [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md)
- [Search](https://xdive.dev/docs/search.md) and [Research](https://xdive.dev/docs/research.md)

---

Source: https://xdive.dev/docs/langchain · All docs for agents: https://xdive.dev/llms.txt


---

# LlamaIndex

`XDiveReader` loads a web page, or every page of a crawled site, into LlamaIndex `Document`s with the page's Markdown as text. Use it to build an index over documentation, a knowledge base or any site, without writing fetching or HTML-cleaning code.

## Install

```bash
pip install "xdive[llama-index]"   # installs llama-index-core
```

The reader is part of the Python SDK and reads `XDIVE_API_KEY` (and `XDIVE_BASE_URL`) like the client. Scrape mode needs a key with the `scrape` scope, crawl mode the `crawl` scope.

## Load a page or a site

```python
from xdive.llama_index import XDiveReader

reader = XDiveReader()

documents = reader.load_data("https://example.com/pricing")   # one page
documents = reader.load_data(
    "https://docs.example.com",
    mode="crawl",
    max_pages=100,
    include_paths=["^/guides/"],
)

for document in documents:
    print(document.metadata["url"], document.metadata["title"], len(document.text))
```

`XDiveReader(client=None)` takes an optional `XDive` client; by default it creates `XDive()` from the environment. Pass your own client for a self-hosted server or other settings:

```python
from xdive import XDive
from xdive.llama_index import XDiveReader

reader = XDiveReader(XDive(api_key="ck_live_…", base_url="https://your-host/api"))
```

`load_data(url, mode="scrape", **options)` returns a `list[Document]`:

| Argument | Description |
|---|---|
| `url` | The page to scrape, or the seed URL of the crawl. |
| `mode` | `"scrape"` loads the one URL. `"crawl"` crawls from it and loads every page that did not fail. Any other value raises `ValueError`. |
| `**options` | Passed to `XDive.scrape` or `XDive.crawl`: crawl parameters such as `max_pages`, `max_depth`, `include_paths` and `exclude_paths` (see [Crawl](https://xdive.dev/docs/crawl.md)), and the client's `timeout` and `poll_interval` for crawls. |

Each document's `text` is the page's Markdown, and its `metadata` is `url` and `title`. Markdown is always requested, so do not pass `formats`.

Crawl mode waits until the crawl finishes before returning; the client's default wait is 600 seconds (`timeout`), after which `CrawlTimeout` is raised and the crawl keeps running on the server. API errors raise the [SDK's exceptions](https://xdive.dev/docs/sdk-python.md#errors).

## Build an index

```python
from llama_index.core import VectorStoreIndex

from xdive.llama_index import XDiveReader

documents = XDiveReader().load_data("https://docs.example.com", mode="crawl", max_pages=200)
index = VectorStoreIndex.from_documents(documents)

answer = index.as_query_engine().query("How do I rotate an API key?")
print(answer)
```

`VectorStoreIndex` embeds the documents and answers queries with the embedding model and LLM configured in LlamaIndex's `Settings`; those calls are billed by their providers, not by XDive.

## Usage

Scrape mode counts one page. Crawl mode counts every page in the crawl's results, including pages that failed and were left out of the documents. To see which pages failed, crawl with the client directly and call `crawl_errors` with the crawl id. See [Usage and billing](https://xdive.dev/docs/usage.md).

## Related

- [LangChain](https://xdive.dev/docs/langchain.md)
- [Python SDK](https://xdive.dev/docs/sdk-python.md)
- [Crawl](https://xdive.dev/docs/crawl.md)

---

Source: https://xdive.dev/docs/llamaindex · All docs for agents: https://xdive.dev/llms.txt


---

# Errors and limits

How the API reports failures, what each status code and error code means, and the rate limits, concurrency limits and monthly quotas that apply to every organization. Read this before putting an integration into production.

## Error format

Every error is JSON with the same fields:

```json
{
  "success": false,
  "code": "CONCURRENCY_LIMIT",
  "error": "too many concurrent requests for this organization (plan limit 5); retry shortly",
  "retryable": true,
  "docs_url": "https://xdive.dev/docs/errors#error-codes"
}
```

| Field | Meaning |
|---|---|
| `code` | A stable identifier. Specific failures have their own code (see [Error codes](#error-codes)); every other error gets the code for its status, such as `INVALID_REQUEST`, `UNAUTHORIZED`, `QUOTA_EXCEEDED` or `NOT_FOUND`. |
| `error` | A message written for people. It may change; do not parse it. |
| `retryable` | `true` when sending the same request again later can succeed: rate limits, overload, timeouts and failures of a site or service the request depends on. `false` means the request, the credentials or the account must change first. |
| `docs_url` | This page. |

Branch on the HTTP status and on `code`, and retry only when `retryable` is `true`, after `Retry-After` seconds when the header is present.

Every response carries an `X-Request-ID` header (`req_…`). Include it when you report a problem.

A few endpoints add detail to the error body:

- `POST /orgs/{id}/batch/scrape` with malformed URLs returns `400` with an `invalid_urls` array, unless you pass `ignore_invalid_urls`.
- `POST /orgs/{id}/extract` returns `502` with a `sources` array giving each URL's error when no source could be read.
- The OAuth endpoints use the OAuth shape, `{"error": "invalid_grant", "error_description": "…"}`, and the MCP endpoint returns JSON-RPC errors.

Crawls and batch scrapes can succeed while some of their pages fail. Each failed page in the results has an `error` message and, for failures you may want to branch on, an `error_code` such as `SITE_RESTRICTED` or `PROXY_NOT_ENABLED`.

## HTTP status codes

| Status | Meaning in this API |
|---|---|
| `200` | Success. |
| `201` | Created (an API key, a browser session). |
| `202` | Accepted: a crawl or batch scrape was queued. Poll its id for the result. |
| `204` | Success with no body (revocations and deletions). |
| `304` | The crawl has not changed since the `ETag` you sent in `If-None-Match`. |
| `400` | The request is malformed or a parameter is invalid. The message says which. |
| `401` | No credentials, or the credentials are invalid, expired, used from an address outside the key's allowlist, or belong to another organization. |
| `402` | The organization's monthly quota for this kind of work is used up. See [Quotas](#quotas). |
| `403` | Authenticated but not allowed: the key lacks a scope, the user is not a member of the organization, the organization is suspended, the account is pending or rejected, or the target page is refused (robots.txt, `SITE_RESTRICTED`, `PROXY_NOT_ENABLED`). |
| `404` | The resource does not exist, or the work found nothing (`NO_SOURCES`, `SCRAPE_LOCKDOWN_CACHE_MISS`). |
| `405` | `GET` or `DELETE` on the MCP endpoint, which accepts `POST` only. |
| `409` | Conflict with the current state: cancelling a finished crawl, exporting an unfinished one, no free browser session. |
| `410` | The browser session has expired. |
| `415` | `POST /orgs/{id}/parse` was not sent as `multipart/form-data`. |
| `422` | The content could not be processed, for example a document that cannot be parsed. |
| `429` | Too many requests. Wait for `Retry-After` seconds. See [Rate limits and concurrency](#rate-limits-and-concurrency). |
| `500` | An unexpected server error. |
| `501` | The feature is not configured on this server: a language model, browser automation or billing. |
| `502` | Something the request depends on failed: the target site, a search engine or the language model. |
| `503` | The server is at capacity (`SERVER_BUSY`) or a dependency is unavailable, such as the job queue or an unconfigured search engine. |
| `504` | The work did not finish in time (`RESEARCH_TIMEOUT`). |

## Error codes

| Code | Status | Returned by | Meaning |
|---|---|---|---|
| `CONCURRENCY_LIMIT` | 429 | Synchronous endpoints | The organization already has its plan's maximum number of synchronous requests running. Sent with `Retry-After: 1`. |
| `SERVER_BUSY` | 503 | Synchronous endpoints | This API server is at capacity and its short wait queue is full. Sent with `Retry-After: 2`. |
| `PROXY_NOT_ENABLED` | 403 | Scrape; `error_code` on crawl pages | The site blocks data-centre networks, and the residential proxy that can reach it is not enabled for your organization, or your organization has used its monthly proxy allowance. The platform administrator enables the proxy and sets the allowance per organization. |
| `SITE_RESTRICTED` | 403 | Scrape; `error_code` on crawl pages | Your organization may use the residential proxy, but the proxy provider refused this destination. Providers typically refuse government and banking sites until the platform's proxy account is identity-verified. Retrying will not help; the platform administrator can see more in the audit log. |
| `SCRAPE_LOCKDOWN_CACHE_MISS` | 404 | Scrape with `lockdown` | Lockdown allows cache reads only, and no cached copy of the page exists. |
| `NO_SOURCES` | 404 | Research | No readable sources were found for the question. |
| `RESEARCH_TIMEOUT` | 504 | Research | The run did not finish in time (45 seconds for quick, 150 for deep). Try `depth: "quick"` or a narrower question. |
| `ACCOUNT_PENDING_APPROVAL` | 403 | Sign-in, sessions, OAuth, MCP | The account is waiting for an administrator's approval. |
| `ACCOUNT_REJECTED` | 403 | Sign-in, sessions, OAuth, MCP | The account's access request was not approved. |
| `EMAIL_NOT_VERIFIED` | 403 | Sign-in | An administrator address listed in `SUPER_ADMIN_EMAILS` has not been verified yet; a new verification link is emailed. |
| `SIGNED_WEBHOOK_REQUIRED` | 400 | Crawl, batch scrape, schedules | A `webhook_url` was given without a `webhook_secret` while the server requires signed webhooks (the default). |
| `WEBHOOK_TARGET_DENIED` | 400 | Crawl, batch scrape, schedules | The webhook URL is not an absolute public `http(s)` URL, for example it points at a private address. |
| `CORS_ORIGIN_DENIED` | 403 | Any endpoint called from a browser | The page's origin is not on the server's allowlist. |
| `RATE_LIMITED` | 429 | Any endpoint | The per-client or per-organization request rate was exceeded. Wait for `Retry-After` seconds. |
| `QUOTA_EXCEEDED` | 402 | Work-creating endpoints | The organization's monthly quota, or the API key's own monthly budget, is used up. The message says which. |
| `INVALID_IDEMPOTENCY_KEY` | 400 | Work-creating `POST` endpoints | The `Idempotency-Key` header is longer than 255 characters or has characters outside printable ASCII. |
| `IDEMPOTENCY_KEY_REUSED` | 422 | Work-creating `POST` endpoints | The key was already used with a different request (another path or body). Use a new key for a new request. |
| `IDEMPOTENCY_KEY_IN_PROGRESS` | 409 | Work-creating `POST` endpoints | The first request with this key is still running. Retry after `Retry-After` seconds to receive its response. |

## Rate limits and concurrency

Four limits protect the service. Each answers with `429` or `503` and a `Retry-After` header in seconds.

| Limit | Applies to | Over the limit |
|---|---|---|
| Per client | Every request: by default 20 per second per API key, or per IP address for requests without a key. | `429` `RATE_LIMITED`, `Retry-After: 1` |
| Requests per minute | Work-creating requests, counted per organization. | `429` `RATE_LIMITED`, with `Retry-After` until the minute resets |
| Concurrent requests | Synchronous requests running at once, per organization. | `429` `CONCURRENCY_LIMIT`, `Retry-After: 1` |
| Server capacity | Synchronous requests running on one API server. | `503` `SERVER_BUSY`, `Retry-After: 2` |

**Requests per minute** counts the calls that create work: scrape, crawl, batch scrape, map, search, research, extract, llms.txt, parse, connectors, and creating or driving browser sessions. Status polling, cancellation and other reads are never counted, so polling a running crawl cannot use up the budget. The window is a fixed minute. These responses carry the budget in the standard `RateLimit-Limit`, `RateLimit-Remaining` and `RateLimit-Reset` headers (seconds until the window resets), and under the same names with an `X-` prefix for older clients. Pace requests from `RateLimit-Remaining` rather than waiting for a `429`.

**Concurrent requests** and **server capacity** apply to the endpoints that fetch while you wait: scrape, map, search, research, extract, llms.txt, parse and connectors. Crawls and batch scrapes are queued instead and never rejected for concurrency: while other organizations' jobs are waiting, an organization gets its plan's fair share of crawl workers, and with nothing else queued it can use more.

Each API server runs up to 128 synchronous requests at once and lets 1,024 more wait up to 5 seconds before answering `SERVER_BUSY`. These are defaults that self-hosted servers can change.

| Plan | Requests per minute | Concurrent synchronous requests | Concurrent crawls (fair share) |
|---|---|---|---|
| `free` | 60 | 5 | 2 |
| `pro` | 600 | 50 | 10 |
| `enterprise` | 6,000 | 500 | 50 |

## Retrying safely with Idempotency-Key

A timeout or a dropped connection does not tell you whether the request ran. Retrying a crawl or a research call blindly can start it twice and use the quota twice. Send an `Idempotency-Key` header on work-creating `POST` requests to make retries safe:

```bash
curl -X POST "$XDIVE_URL/api/orgs/$ORG_ID/crawl" \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Idempotency-Key: 6f1c2d4e-8a7b-4c3d-9e0f-1a2b3c4d5e6f" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_depth": 1}'
```

- Use a new random value, such as a UUID, for each new request, and the same value for every retry of it.
- For 24 hours, a retry with the same key and the same body returns the first response, with the header `Idempotent-Replayed: true`, without running the request or counting it again.
- A retry while the first request is still running gets `409` `IDEMPOTENCY_KEY_IN_PROGRESS` with `Retry-After`.
- The same key with a different path or body gets `422` `IDEMPOTENCY_KEY_REUSED`.
- Rate-limit (`429`) and server (`5xx`) responses are not stored, so a retry after them runs the request. Responses larger than 4 MB are delivered but not stored.
- Keys belong to the organization. Two organizations can use the same value without effect on each other.

## Quotas

Each plan has monthly quotas, counted per calendar month in UTC (see [Usage and billing](https://xdive.dev/docs/usage.md) for what counts):

| Plan | Pages per month | Searches per month | Browser seconds per month | Browser sessions at once | Schedules |
|---|---|---|---|---|---|
| `free` | 1,000 | 100 | 600 | 1 | 3 |
| `pro` | 50,000 | 10,000 | 36,000 | 3 | 50 |
| `enterprise` | Unlimited | Unlimited | Unlimited | 10 | Unlimited |

When a quota is used up, requests that would use more of it return `402`:

```json
{ "error": "monthly page quota for the free plan reached (1000 pages); upgrade to continue" }
```

Quotas are checked before work starts, so a crawl that starts under the limit runs to completion even if it ends above it. Creating a schedule over the plan's cap also returns `402`; starting a browser session when the plan's sessions are all in use returns `409`.

## Retries

- **`429` and `503`:** wait for `Retry-After` seconds, then retry.
- **`402`:** retrying will not help until the next month or a plan upgrade.
- **Other `4xx`:** fix the request; the same request fails again.
- **`502` and `504`:** a dependency failed or was slow. A retry may succeed, but a successful retry is metered like any other request.

### How the SDKs retry

Both SDKs retry `429` and `503` automatically. Each retry waits the response's `Retry-After` (1 second when it is missing), up to `max_retries` (Python) or `maxRetries` (JavaScript) times, default `3`. After that they raise `RateLimited` / `RateLimitedError`, which carries the wait the server asked for and the error `code`.

| Situation | Python | JavaScript |
|---|---|---|
| Still throttled after retries (`429`, `503`) | `RateLimited` (`retry_after`) | `RateLimitedError` (`retryAfterSeconds`) |
| Quota used up (`402`) | `QuotaExceeded` | `QuotaExceededError` |
| Any other error status | `XDiveError` | `XDiveError` |
| Crawl not finished within the wait timeout | `CrawlTimeout` | `CrawlTimeoutError` |

All of them extend `XDiveError`, which exposes `status`, `code` and the message.

**Python**

```python
from xdive import XDive, XDiveError, QuotaExceeded, RateLimited

xdive = XDive(max_retries=5)
try:
    page = xdive.scrape("https://example.com")
except RateLimited as error:
    print(f"still throttled ({error.code}); retry in {error.retry_after}s")
except QuotaExceeded as error:
    print("quota used up:", error.message)
except XDiveError as error:
    print(error.status, error.code, error.message)
```

**JavaScript**

```ts
import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive";

const xdive = new XDive({ maxRetries: 5 });
try {
  const page = await xdive.scrape("https://example.com");
} catch (error) {
  if (error instanceof RateLimitedError) {
    console.log(`still throttled (${error.code}); retry in ${error.retryAfterSeconds}s`);
  } else if (error instanceof QuotaExceededError) {
    console.log("quota used up:", error.message);
  } else if (error instanceof XDiveError) {
    console.log(error.status, error.code, error.message);
  } else {
    throw error;
  }
}
```

The SDKs retry every `429` and `503`, including a `503` from a server whose search engine is not configured, so such a call fails only after the retries.

---

Source: https://xdive.dev/docs/errors · All docs for agents: https://xdive.dev/llms.txt


---

# Usage and billing

Usage is counted per organization and per calendar month (UTC), and compared with the monthly quotas of the organization's plan. This page lists what each endpoint counts, what is free, how to read the counters, and the limits of each plan.

## What is metered

The meter keeps four counters per organization and month:

| Counter | Measures | Quota |
|---|---|---|
| `pages` | Pages fetched or documents parsed | Pages per month |
| `searches` | Search engine queries | Searches per month |
| `browserSeconds` | Time spent in an interactive browser | Browser seconds per month |
| `crawls` | Crawl and batch scrape jobs started | None (informational) |

What each endpoint adds:

| Endpoint | Counted | When |
|---|---|---|
| [Scrape](https://xdive.dev/docs/scrape.md) | 1 page | When the page is fetched, from the site or from the cache. |
| Scrape with `actions`, `wait_for` or `screenshot` | Browser seconds, not pages | When the browser run ends. |
| [Crawl](https://xdive.dev/docs/crawl.md) | 1 crawl, then 1 page per page in the results | The crawl when it is queued; the pages when the job finishes, including pages that failed. |
| [Batch scrape](https://xdive.dev/docs/batch-scrape.md) | 1 crawl, then 1 page per URL in the batch | The same as crawl. |
| [Search](https://xdive.dev/docs/search.md) | 1 search | When the search engines answer, including when the answer comes from the search cache. |
| Search with `search_depth: "advanced"` | 1 page per result read | After the results are read. |
| Search with `scrape` or `include_answer` | 1 page per result scraped | After the results are scraped. |
| [Research](https://xdive.dev/docs/research.md) | The `searches` and `pages_read` reported in the response's `usage` | When a new run succeeds. |
| [Extract](https://xdive.dev/docs/extract.md) | 1 page per source read | After the sources are read, before the model runs. |
| [llms.txt](https://xdive.dev/docs/llms-txt.md) | 1 page per page scraped | After the pages are scraped. |
| `POST /orgs/{id}/parse` | 1 page per document | When parsing succeeds. |
| Browser session actions | Browser seconds | Per actions call: its duration rounded up to whole seconds, at least 1. |

### What is free

- **Cached research answers.** Asking the same research question with the same options again within 10 minutes returns the stored answer with `"cached": true`, and it is not metered. The cache is kept by each API server, so on a deployment with several servers a repeat can still run (and be metered) again. Pass `"no_cache": true` to force a new, metered run.
- **Map.** [Map](https://xdive.dev/docs/map.md) is not metered, but it is refused with `402` once the page quota is used up.
- **Reads and polling.** Crawl status, cancellation, crawl errors and `GET /whoami` are never metered.
- **Rejected and failed requests.** A request refused before any work starts (validation, scope, rate limit or quota) costs nothing, and so do a scrape whose page could not be fetched, a search the engines did not answer, and a research run that failed or timed out.

> **Note:**
>
> Cache hits on scrape and search are faster and do not touch the target site or the search engine, but they are still counted: a cached scrape counts one page and a cached search counts one search.

## Reading usage

The dashboard shows the current period under **Billing** (**Billing & usage**). The same numbers come from the API:

`GET https://xdive.dev/api/orgs/{id}/usage`

This endpoint takes a [session token](https://xdive.dev/docs/authentication.md#dashboard-sessions) from a member of the organization; API keys cannot read usage. `GET /whoami` with an API key returns the organization's `plan`.

```bash
curl "https://xdive.dev/api/orgs/$ORG_ID/usage" \
  -H "Authorization: Bearer $SESSION_TOKEN"
```

```json
{
  "browserSecondsUsed": 45,
  "crawlsUsed": 4,
  "limits": {
    "name": "free",
    "monthlyPages": 1000,
    "monthlySearches": 100,
    "monthlyBrowserSeconds": 600,
    "maxBrowserSessions": 1,
    "maxSchedules": 3,
    "requestsPerMinute": 60,
    "maxConcurrentRequests": 5,
    "maxConcurrentCrawls": 2
  },
  "pagesUsed": 312,
  "period": "2026-10",
  "plan": "free",
  "proxy": {
    "enabled": false,
    "monthlyPages": 200,
    "pagesUsed": 0
  },
  "schedules": 1,
  "searchesUsed": 27
}
```

| Field | Type | Description |
|---|---|---|
| `period` | string | The current period, the UTC calendar month as `YYYY-MM`. |
| `plan` | string | The organization's plan. |
| `limits` | object | The plan's limits; see [Plans](#plans). `-1` means unlimited. |
| `pagesUsed` | integer | Pages counted this period. |
| `searchesUsed` | integer | Searches counted this period. |
| `browserSecondsUsed` | integer | Browser seconds counted this period. |
| `crawlsUsed` | integer | Crawl and batch scrape jobs started this period. |
| `schedules` | integer | Schedules that exist now (a count, not monthly usage). |
| `proxy` | object | The organization's residential proxy entitlement, used for [government and banking sites](https://xdive.dev/docs/scrape.md#government-and-banking-sites): `enabled` (set by the platform administrator; `false` until then), `monthlyPages` (the monthly allowance of proxied pages; `0` allows none) and `pagesUsed` (pages fetched through the proxy this period). |

Counters are updated within seconds of the work, not instantly.

## Per-key budgets

An API key can have its own monthly page and search budget, on top of the plan quota. Give a key with a budget to an AI agent or a contractor, and it stops at that budget without using up the organization's quota.

```bash
curl -X POST "$XDIVE_URL/api/orgs/$ORG_ID/api-keys" \
  -H "Authorization: Bearer $SESSION_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "research agent", "scopes": ["scrape", "search"], "monthly_page_limit": 500, "monthly_search_limit": 100}'
```

- `0`, the default, means only the plan quota applies.
- Change a budget with `PATCH /orgs/{id}/api-keys/{keyId}` and `{"monthly_page_limit": 1000}`. An omitted field keeps its value. The change applies at once.
- `GET /orgs/{id}/api-keys` lists each key's `pagesUsed` and `searchesUsed` this month.
- A request over the key's budget gets `402` `QUOTA_EXCEEDED`, and the message names the key's budget. A crawl started with less budget left than its `max_pages` is shortened to what is left. A batch scrape with more URLs than the budget allows is refused.
- Budgets reset with the plan period, at the start of each calendar month (UTC).

## Plans

Every organization is on one plan. New organizations start on `free`.

| Limit | `free` | `pro` | `enterprise` |
|---|---|---|---|
| Pages per month (`monthlyPages`) | 1,000 | 50,000 | Unlimited |
| Searches per month (`monthlySearches`) | 100 | 10,000 | Unlimited |
| Browser seconds per month (`monthlyBrowserSeconds`) | 600 | 36,000 | Unlimited |
| Browser sessions at once (`maxBrowserSessions`) | 1 | 3 | 10 |
| Schedules (`maxSchedules`) | 3 | 50 | Unlimited |
| Requests per minute (`requestsPerMinute`) | 60 | 600 | 6,000 |
| Concurrent synchronous requests (`maxConcurrentRequests`) | 5 | 50 | 500 |
| Concurrent crawls, fair share (`maxConcurrentCrawls`) | 2 | 10 | 50 |

When a monthly quota is used up, the requests that need it return `402` until the next month or an upgrade. See [Errors and limits](https://xdive.dev/docs/errors.md#quotas) for the details and for how the per-minute and concurrency limits behave.

## Changing plans

Only the organization's owner can change its plan, from **Billing** in the dashboard.

- When the server has billing configured, upgrading goes through a hosted checkout (`POST /orgs/{id}/billing/checkout` with `{"plan": "pro"}` returns its `url`), and the new plan applies once the payment is confirmed. Downgrading to `free` is immediate.
- When billing is not configured, as on many self-hosted servers, the owner switches plans directly with `PUT /orgs/{id}/plan` and `{"plan": "pro"}`.

---

Source: https://xdive.dev/docs/usage · All docs for agents: https://xdive.dev/llms.txt


---

# Self-hosting

The whole platform (API, crawl workers, dashboard and these docs) is in one repository and runs on your own servers with Docker Compose. This page covers the essentials; the repository guides have the full detail:

- [RUNNING.md](https://github.com/sadeghianme/Crawler/blob/main/RUNNING.md): step-by-step local setup, with Docker or from source.
- [README.md](https://github.com/sadeghianme/Crawler/blob/main/README.md): features, configuration and deployment notes.
- [OPERATIONS.md](https://github.com/sadeghianme/Crawler/blob/main/OPERATIONS.md): production deploys, rollback, backups, monitoring and alerts.
- [SCALING.md](https://github.com/sadeghianme/Crawler/blob/main/SCALING.md): capacity settings and running several API and worker nodes.

## What runs

| Service | Role |
|---|---|
| `server` | The API. Listens on port 8080. |
| `worker` | Runs crawls and batch scrapes from the queue. Includes headless Chromium. Scale it for throughput. |
| `web` | The dashboard and docs. Its nginx forwards `/api/` to the API with the prefix removed. |
| `postgres`, `pgbouncer` | Users, organizations, keys, usage, crawl history. |
| `redis` | Job queue, page cache, rate limits and shared usage counters. |
| `minio` | S3-compatible storage for crawl results. |

## Run it with Docker Compose

```bash
git clone https://github.com/sadeghianme/Crawler.git
cd Crawler
cp .env.example .env   # then set JWT_SECRET and SECRETS_KEY to long random strings
docker compose up --build
```

Open http://localhost:8088 and sign up. The API is also reachable directly at http://localhost:8080. Without SMTP settings, verification and password-reset emails are written to the server log (`docker compose logs server`). Add crawl workers with `docker compose up --scale worker=4`.

For production, the repository layers `docker-compose.prod.yml` over the base file (`docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d`). It adds memory limits and restart policies, removes host port bindings and expects a reverse proxy on an external `web_proxy` network. It is the hosted service's own configuration (for example, `APP_URL` is set to the hosted domain), so review it before reusing it. [OPERATIONS.md](https://github.com/sadeghianme/Crawler/blob/main/OPERATIONS.md) describes the deploy and rollback scripts built around it.

## Configuration

Settings are environment variables, documented in `.env.example`. The ones most deployments need:

| Variable | Purpose |
|---|---|
| `JWT_SECRET` | **Required.** Signs dashboard sessions; use the same value on every API node. A server started without it uses a temporary secret, so sessions end at every restart, and `docker-compose.yml` falls back to a development placeholder. |
| `SECRETS_KEY` | Encrypts stored secrets such as organizations' model keys. Set a stable value in production; falls back to `JWT_SECRET`. |
| `APP_URL` | The public URL of the dashboard. Used in email links and payment redirects, as the OAuth issuer for MCP clients, and as an allowed browser origin. |
| `DATABASE_URL` | PostgreSQL connection string. Without it a source build uses a local SQLite file. |
| `REDIS_ADDR` | Redis for the queue, cache, rate limits and usage counters shared across nodes. In-memory when unset. |
| `R2_ENDPOINT`, `R2_ACCESS_KEY`, `R2_SECRET_KEY`, `R2_BUCKET`, `R2_USE_SSL` | S3-compatible storage (MinIO, Cloudflare R2, S3) for crawl results. Results stay in the database when unset. |
| `SMTP_HOST`, `SMTP_PORT`, `SMTP_USER`, `SMTP_PASSWORD`, `SMTP_FROM` | Email for verification, password reset and approval notices. |
| `LLM_API_KEY`, `LLM_BASE_URL`, `LLM_MODEL` | The server's default OpenAI-compatible model, used by research, extract, `include_answer` and model-based formats such as `json`. Organizations can bring their own under **AI Extraction**. |
| `SEARCH_PROVIDER`, `BRAVE_SEARCH_API_KEY`, `SERPER_API_KEY` | Search engines for search and research. `SEARCH_PROVIDER` is a comma-separated list (`brave`, `serper`); the default is `brave`. |
| `METRICS_TOKEN` | Enables and protects the Prometheus `/metrics` endpoint. |
| `STRIPE_SECRET_KEY`, `STRIPE_WEBHOOK_SECRET`, `STRIPE_PRICE_PRO`, `STRIPE_PRICE_ENTERPRISE` | Paid plans through Stripe checkout. Without them, owners switch plans directly. |
| `ORG_RATE_LIMITS` | `off` disables the per-plan requests-per-minute budgets. |
| `ADMISSION_MAX_INFLIGHT`, `ADMISSION_MAX_QUEUE`, `ADMISSION_MAX_WAIT` | Synchronous requests per API node: running (128), waiting (1024) and how long they may wait (`5s`) before `SERVER_BUSY`. |

A super admin can also set the model, the search engines and their keys in the dashboard under **Global Settings**; values stored there take precedence over the environment.

> **Warning: Compose passes only the variables it lists**
>
> The Compose files hand the server and worker only the variables named in their `x-backend-env` block. `docker-compose.yml` does not currently list `SUPER_ADMIN_EMAILS`, `REGISTRATION_APPROVAL` or the search settings, so setting them in `.env` alone has no effect there. Add the ones you use to that block.

## Point the SDKs at your server

Set the base URL to your server's API root, including `/api` when requests go through the `web` container (for the default Compose setup, `http://localhost:8088/api`). The API container itself serves the same routes without the prefix, at `http://localhost:8080`.

**cURL**

```bash
export XDIVE_BASE_URL="https://your-host/api"   # read by both SDKs
curl "$XDIVE_BASE_URL/whoami" -H "X-API-Key: $XDIVE_API_KEY"
```

**Python**

```python
from xdive import XDive

xdive = XDive(base_url="https://your-host/api")  # or set XDIVE_BASE_URL
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive({ baseUrl: "https://your-host/api" }); // or set XDIVE_BASE_URL
```

API keys are issued by your server, in its dashboard, exactly as on the hosted service. The stdio MCP server in the repository (`cmd/mcp`) reads its base URL from `CRAWLER_API_URL`, not from `XDIVE_BASE_URL`, and its key from `CRAWLER_API_KEY`.

## Egress proxy

Some sites refuse all connections from data-centre addresses, which is common for government and banking sites. Set `EGRESS_PROXY_URL` to an HTTP (CONNECT) or SOCKS5 proxy, typically a residential proxy with country targeting in the username:

```bash
EGRESS_PROXY_URL=http://USER-country-in:PASSWORD@gate.provider.com:7000
EGRESS_PROXY_DOMAINS=boi.gov.in,bankofindia.bank.in   # optional: always proxy these
```

Configuring the proxy does not let anyone use it yet. Proxy access is off for every organization, existing and new, until a super admin turns it on under **Global Settings** → **Organizations**, where each organization also gets a monthly cap of proxied pages (200 by default, `0` to allow none, at most 100,000) and shows this month's use. The same settings are available as `PATCH /admin/orgs/{id}/proxy` with `{"enabled": true, "monthly_pages": 500}`, and every change is written to the organization's audit log. Only scrapes, crawls and batch scrapes use the proxy; search, research, extract, map and llms.txt never do.

For an enabled organization, fetches go direct first and use the proxy only when the site cannot be reached or stays blocked; domains in `EGRESS_PROXY_DOMAINS` always use it. Once the organization's cap is reached, its fetches go direct only until the next calendar month. The SSRF guard still applies, so the proxy cannot be used to reach private addresses. Proxied pages report `fetch_route: "proxy"`. A site that blocks data-centre networks returns `403` with `PROXY_NOT_ENABLED` for an organization without access (or past its cap), and `403` with `SITE_RESTRICTED` when the proxy provider refuses the destination.

## Access control

By default anyone can sign up and use the server immediately. Three variables change that:

- **`SUPER_ADMIN_EMAILS`**: a comma-separated list of addresses that become platform super admins. An address is promoted only after its owner verifies it, at verification or at sign-in; until then, signing in returns `403 EMAIL_NOT_VERIFIED` and emails a new link. Super admins see **Global Settings** in the dashboard.
- **Invite-only sign-up**: super admins turn it on under **Global Settings** → **Access requests** → **Sign-up policy**, and keep an allowlist of addresses and `@domain` entries there. Allowlisted addresses are approved as soon as they sign up; everyone else is held as an access request, as with `REGISTRATION_APPROVAL=required`. Changes apply without a restart. `REGISTRATION_ALLOWED_EMAILS` adds entries from the environment and also turns invite-only on.
- **`REGISTRATION_APPROVAL=required`**: new sign-ups are held for review. Sign-up answers `202` with `"status": "pending_approval"`, the addresses in `SUPER_ADMIN_EMAILS` are emailed, and super admins approve or reject requests under **Global Settings** → **Access requests**. The user is emailed when approved; until then signing in returns `403 ACCOUNT_PENDING_APPROVAL`. Rejecting an account ends its sessions and OAuth grants at once and suspends the organizations it is the only member of.

Accounts listed in `SUPER_ADMIN_EMAILS` are approved when their address is verified, so they are never left waiting for another administrator.

---

Source: https://xdive.dev/docs/self-hosting · All docs for agents: https://xdive.dev/llms.txt


---

# Changelog

User-facing changes to the API, the SDKs and the dashboard, newest first. Releases with only internal changes (v1.0.1 and v1.0.2) are left out.

## Unreleased

- SDKs: the packages are now called `xdive`. Install the Python package with `pip install xdive` (import `xdive`, classes `XDive` and `AsyncXDive`) and the JavaScript package with `npm install xdive`, `pnpm add xdive`, `yarn add xdive` or `bun add xdive` (class `XDive`). They read `XDIVE_API_KEY` and `XDIVE_BASE_URL`; the older `CRAWLER_API_KEY` and `CRAWLER_BASE_URL` still work.
- SDKs: every `POST` carries an `Idempotency-Key`, reused by each retry, and timeouts and dropped connections are retried safely.
- New SDKs for [Java](https://xdive.dev/docs/sdk-java.md) (`dev.xdive:xdive`), [.NET](https://xdive.dev/docs/sdk-dotnet.md) (`XDive` on NuGet), [PHP](https://xdive.dev/docs/sdk-php.md) (`xdive/xdive`), [Ruby](https://xdive.dev/docs/sdk-ruby.md) (`xdive`) and [Rust](https://xdive.dev/docs/sdk-rust.md) (`xdive`), with the same methods, typed errors, safe retries and an `Idempotency-Key` on every POST.
- New [Go SDK](https://xdive.dev/docs/sdk-go.md) (`github.com/xdive/xdive-go`) and [Vite plugin](https://xdive.dev/docs/vite-plugin.md) (`vite-plugin-xdive`), which fetches pages as Markdown at build time and can emit an `llms.txt`.
- Docs for AI agents: [`/llms.txt`](https://xdive.dev/llms.txt), [`/llms-full.txt`](https://xdive.dev/llms-full.txt), every docs page as Markdown at `/docs/<slug>.md`, an OpenAPI 3.1 spec at [`/openapi.json`](https://xdive.dev/openapi.json), drop-in `AGENTS.md`, Claude skill and Cursor rule files, and the new [For AI agents](https://xdive.dev/docs/agents.md) page.
- Research: identical requests from the same organization are served from a 10-minute result cache, marked `"cached": true` and not metered; `"no_cache": true` forces a new run.
- Research: quick mode answers once enough sources have arrived instead of waiting for the slowest page, and runs one follow-up search when its sources do not cover the question.
- Research: two-step questions are answered by first resolving the intermediate entity; `usage.hops` and `usage.plan` report what was done.
- Research: sources that could not be read in time contribute their search snippet, marked `"snippet": true`.
- Research: `usage` reports where the time went (`search_ms`, `read_ms`, `model_ms`), and a run no longer returns an empty answer.
- Research: `schema` must be a JSON object.
- Search: `search_depth: "advanced"` keeps the engines' ranking and adds each result's passages and `relevance_score` instead of re-ordering results.
- Search: `include_answer` has its own time budget, and a page read for both advanced search and scraping is counted once.
- Scrape: a site that the egress proxy provider refuses (typically government and banking sites) returns `403` with code `SITE_RESTRICTED` and a plain explanation; crawl pages carry `error_code: "SITE_RESTRICTED"`.
- Self-hosting: an optional egress proxy (`EGRESS_PROXY_URL`, `EGRESS_PROXY_DOMAINS`) for sites that refuse data-centre addresses; proxied pages report `fetch_route: "proxy"`.
- Security: fetches through the egress proxy are checked by the SSRF guard.
- Self-hosting: `REGISTRATION_APPROVAL=required` holds new sign-ups until a super admin approves them; sign-up answers `202` and sign-in returns `ACCOUNT_PENDING_APPROVAL` or `ACCOUNT_REJECTED`.
- Accounts: rejecting an account ends its sessions and OAuth grants at once and suspends the organizations it alone belongs to.
- Accounts: addresses in `SUPER_ADMIN_EMAILS` become super admins only after they are verified; sign-in returns `EMAIL_NOT_VERIFIED` until then.
- Self-hosting: `LLM_MODEL_ROTATION=round_robin` spreads requests across the configured models.
- Dashboard: fixed header and sidebar, and loading placeholders in place of spinners.
- SDKs: the research result types include `cached`, `hops`, snippet sources and phase timings.

## v1.5.0 (2026-10-10)

- New `POST /orgs/{id}/research`: answers (quick) and reports (deep) from the web with verified citations, or structured `data` from a JSON Schema. Also available as an MCP tool.
- Search: several engines (Brave and Serper) are queried together and their rankings merged, results are cached, and near-duplicates are removed.
- Search: `search_depth: "advanced"` returns each result's relevant passages and a `relevance_score`; `auto_parameters` infers sources and freshness from the query; `include_answer` adds a short cited answer.
- New Python SDK (`xdive`, with `XDive` and `AsyncXDive`) and JavaScript SDK (`xdive`), with LangChain tools and a LlamaIndex reader.
- New `GET /whoami`: the organization, scopes and plan of an API key, used by the SDKs to find the organization.
- New public `GET /status` summary for status pages and monitors.
- Language-model calls retry transient failures and fall back to `LLM_FALLBACK_MODELS`; a failing search engine is skipped.

## v1.4.0 (2026-10-09)

- Crawl status responses carry an `ETag`; sending it in `If-None-Match` returns `304` while nothing has changed. Running crawls suggest a polling interval with `Retry-After: 2`.
- Synchronous endpoints enforce a per-organization concurrency limit (`429 CONCURRENCY_LIMIT`) and per-server capacity (`503 SERVER_BUSY`), both with `Retry-After`.
- Crawl workers are shared fairly between organizations by plan, and browser-rendered crawls run in their own queue.
- Synchronous scrape is now checked against the page quota and counts one page per successful scrape.
- `max_age` can accept older cached content as well as require fresher content; simultaneous requests for the same uncached page share one fetch.
- Security: the page cache is keyed by request headers as well as URL, so a page fetched with cookies or authorization headers is no longer served to requests sent without them.
- With billing configured, `PUT /orgs/{id}/plan` only downgrades to `free`; paid plans are activated through checkout.

## v1.3.1 (2026-10-09)

- Security: headless Chrome traffic passes through the SSRF guard.
- Unreachable hosts fail fast with a clear error instead of escalating to the browser, and the scrape `502` message includes the page's error.

## v1.3.0 (2026-10-08)

- MCP clients such as ChatGPT, claude.ai, Claude Code and Codex connect with OAuth 2.1. Connected clients are listed, and can be revoked, under **API Keys** → **Connected apps**.
- Breaking: the MCP route with an API key in the URL (`/api/mcp/{key}`) is removed. Clients that send headers keep using API keys.

## v1.2.1 (2026-10-08)

- Fixed: `POST /api/mcp` no longer answers with a `301` redirect, which broke header-authenticated MCP clients.

## v1.2.0 (2026-10-07)

- MCP connectors for ChatGPT and claude.ai, and a new `fetch` tool that returns a page's full Markdown.
- `GET` and `DELETE` on the MCP endpoint return `405`.
- Redesigned landing and sign-in pages with light and dark themes.

## v1.1.1 (2026-10-07)

- Every default fetch path escalates past WAF bot filters with Chrome TLS impersonation, so protected sites return content instead of `403` or `404`. `adaptive_fetch: false` still sends a single plain request.

## v1.1.0 (2026-10-07)

- A public landing page for signed-out visitors.

## v1.0.3 (2026-10-07)

- Per-organization requests-per-minute budgets by plan (60, 600 and 6,000) on work-creating endpoints, reported in `X-RateLimit-Limit` and `X-RateLimit-Remaining`. Self-hosted servers can turn them off with `ORG_RATE_LIMITS=off`.

## v1.0.0 (2026-10-07)

First release.

- Scrape in many formats (Markdown, HTML, links, text, screenshots, JSON and more), crawl with webhooks, batch scrape, map, extract, llms.txt, document parsing, and web, news and image search.
- Adaptive fetching (plain HTTP, TLS impersonation, headless browser), interactive browser sessions, PII redaction, and zero-data-retention and lockdown modes.
- Organizations, API keys with scopes, expiry and IP allowlists, usage metering with plan quotas, Stripe billing and an audit log.
- Schedules with change monitoring, an MCP server (stdio and Streamable HTTP), the dashboard, and self-hosting with Docker Compose.

---

Source: https://xdive.dev/docs/changelog · All docs for agents: https://xdive.dev/llms.txt