For AI agents

This page is for AI agents and the people who build them: which endpoint or MCP tool fits which job, how to authenticate, what things cost, and how to retry safely. Every docs page is also available as plain Markdown, so an agent can read the docs without rendering this site.

Machine-readable docs

Resource	URL	Contents
`llms.txt`	https://xdive.dev/llms.txt	An index of every docs page in the llmstxt.org format, linking to the Markdown versions.
`llms-full.txt`	https://xdive.dev/llms-full.txt	All docs pages in one Markdown file.
Markdown pages	`https://xdive.dev/docs/<slug>.md`	Each page as Markdown, for example scrape.md. Docs pages advertise theirs with `<link rel="alternate" type="text/markdown">` and a `Link` header.
OpenAPI	https://xdive.dev/openapi.json	An OpenAPI 3.1 description of the public REST API: every endpoint, its parameters, responses, scopes and error codes. Generate a client or tool definitions from it.
MCP server	`https://xdive.dev/api/mcp`	The hosted MCP server (Streamable HTTP).

Choose the endpoint or tool

Pick the cheapest call that answers the need. MCP tools call the same API, so the same limits and metering apply.

Job	REST endpoint	MCP tool
Find pages about a topic	`POST /orgs/{id}/search`	`search`
Recent news	`POST /orgs/{id}/search` with `sources: ["news"]` and `freshness`	`search` with `sources: ["news"]`
Read one known URL as Markdown	`POST /orgs/{id}/scrape`	`fetch` (or `scrape` for other formats)
A page as HTML, links, a screenshot or structured JSON	`POST /orgs/{id}/scrape` with `formats`	`scrape` with `formats`
A cited answer to a question	`POST /orgs/{id}/research`	`research`
Fields or tables from up to 10 pages	`POST /orgs/{id}/extract`	`extract`
Find the right URL on a site before reading it	`POST /orgs/{id}/map` with `search`	`map`
Read many pages of one site	`POST /orgs/{id}/crawl`, then `GET /orgs/{id}/crawls/{crawlId}`	`crawl`, then `crawl_status`
A known list of up to 100 URLs	`POST /orgs/{id}/batch/scrape`	(none; use `fetch` per URL)
An `llms.txt` for someone else's site	`POST /orgs/{id}/llmstxt`	(none)
Convert an uploaded PDF, DOCX or XLSX	`POST /orgs/{id}/parse` (see the OpenAPI spec)	(none)
Click, type and scroll in a real browser	`POST /orgs/{id}/scrape` with `actions`; interactive sessions under `/orgs/{id}/browser/sessions` (see the OpenAPI spec)	(none)

Rules of thumb:

Prefer search then scrape (or fetch) when you need to quote precisely; prefer research when the user wants an answer, not documents.
Do not crawl a site to read one page: map it with search, then scrape the page you need.
Keep crawls small and targeted with max_pages, max_depth and include_paths.
Crawl and batch scrape answer 202 with a job id. Poll GET /orgs/{id}/crawls/{crawlId}, honouring Retry-After and ETag (details), or pass webhook_url.

Authentication

There are three ways for an agent to authenticate. See Authentication for the details.

API key (REST and MCP). Send Authorization: Bearer ck_live_… or X-API-Key: ck_live_…. A key belongs to one organization; call GET /whoami once to get its orgId and build /orgs/{orgId}/… paths from it. Keys carry scopes (scrape, crawl, search, browser, connectors); new keys get scrape and crawl, so add search for search and research. Keep keys in environment variables, never in prompts, code or logs.
Device sign-in (REST and MCP), for an agent or CLI without a key. Call POST /api/oauth/device_authorization, show the person the verification_uri and the user_code, and poll POST /api/oauth/token every interval seconds. Once the person approves at /device, the token response carries an API key and its org_id. See Device sign-in.
OAuth (MCP only). MCP clients such as Claude, ChatGPT, Cursor and Codex sign in with OAuth 2.1 and PKCE: paste the server URL, sign in, choose the organization, approve. No key is copied. OAuth tokens work only on /api/mcp, not on the REST API.

API keys cannot manage the organization: API keys, usage, schedules, history and exports need a dashboard session.

Costs and limits

Usage is metered per organization and month in pages, searches and browser seconds. Map, status polling, cancellation and GET /whoami are free, and so are requests that are rejected before any work starts. See Usage and billing for what each endpoint counts, and Errors and limits for the monthly quotas. A used-up quota answers 402; that is something to tell the user, not to retry.

If a person gives you an API key, it may have its own monthly page and search budget, set so an agent cannot spend the whole organization's quota. Over that budget the API also answers 402 QUOTA_EXCEEDED, and the message names the key's budget. See Per-key budgets.

Synchronous calls (scrape, map, search, research, extract, llms.txt, parse) time out on the server: about 30 seconds for a scrape, longer for deep research. Set client timeouts to at least 120 seconds, and 180 seconds for depth: "deep" research.

Rate limits

Each limit answers 429 (or 503 when a server is at capacity) with a Retry-After header in seconds. See Rate limits and concurrency for the numbers per plan.

Work-creating requests also report the per-minute budget in standard headers:

Header	Meaning
`RateLimit-Limit`	Requests allowed in the current window.
`RateLimit-Remaining`	Requests left in the current window.
`RateLimit-Reset`	Seconds until the window resets.

The older X-RateLimit-Limit and X-RateLimit-Remaining headers carry the same numbers. Slow down when RateLimit-Remaining gets close to zero instead of waiting for a 429.

Retries and idempotency

429 and 503: wait Retry-After seconds (1 when it is missing), then retry. Cap the number of retries.
502 and 504: a dependency failed or was slow. One retry may succeed, and is metered like any request.
402, 401, 403 and other 4xx: do not retry the same request. Fix it or report it.
Every error body has retryable: retry only when it is true. Branch on the HTTP status and the error body's code (for example CONCURRENCY_LIMIT), never on the error text. Include the X-Request-ID response header when you report a problem.

To retry a POST safely, send an Idempotency-Key header with a unique value (a UUID) per logical request, and reuse it on every retry of that request:

bash

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: 7f0c2a9e-3b1d-4c55-9a7e-2d4f6b8c1e03" \
  -d '{"url": "https://docs.example.com", "max_pages": 50}'

The same key with the same body within 24 hours returns the stored response, with the header Idempotent-Replayed: true, instead of doing the work (and metering it) again. A retried crawl therefore does not start a second crawl.
The same key with a different body is rejected with 422 and the code IDEMPOTENCY_KEY_REUSED. Use a new key for a new request.

Instructions for your agent

Drop-in instruction files that teach a coding agent to use the API correctly:

File	For	Install
AGENTS.md	Codex, Cursor, Copilot, Gemini CLI and other agents that read `AGENTS.md`	Append it to the `AGENTS.md` at the root of your repository.
SKILL.md	Claude Code and the Claude Agent SDK	Save it as `.claude/skills/xdive-api/SKILL.md` in your repository (or `~/.claude/skills/xdive-api/SKILL.md`).
xdive.mdc	Cursor	Save it as `.cursor/rules/xdive.mdc`.

bash

curl -fsSL https://xdive.dev/agents/AGENTS.md >> AGENTS.md
mkdir -p .claude/skills/xdive-api && curl -fsSL -o .claude/skills/xdive-api/SKILL.md https://xdive.dev/agents/xdive-api/SKILL.md
mkdir -p .cursor/rules && curl -fsSL -o .cursor/rules/xdive.mdc https://xdive.dev/agents/xdive.mdc

For assistants that use the tools directly rather than writing code against the API, connect the MCP server; its plugin for Claude Code and Codex adds a web-research skill on top of the same tools.