# For AI agents This page is for AI agents and the people who build them: which endpoint or MCP tool fits which job, how to authenticate, what things cost, and how to retry safely. Every docs page is also available as plain Markdown, so an agent can read the docs without rendering this site. ## Machine-readable docs | Resource | URL | Contents | |---|---|---| | `llms.txt` | [https://xdive.dev/llms.txt](https://xdive.dev/llms.txt) | An index of every docs page in the [llmstxt.org](https://llmstxt.org) format, linking to the Markdown versions. | | `llms-full.txt` | [https://xdive.dev/llms-full.txt](https://xdive.dev/llms-full.txt) | All docs pages in one Markdown file. | | Markdown pages | `https://xdive.dev/docs/.md` | Each page as Markdown, for example [scrape.md](https://xdive.dev/docs/scrape.md). Docs pages advertise theirs with `` and a `Link` header. | | OpenAPI | [https://xdive.dev/openapi.json](https://xdive.dev/openapi.json) | An OpenAPI 3.1 description of the public REST API: every endpoint, its parameters, responses, scopes and error codes. Generate a client or tool definitions from it. | | MCP server | `https://xdive.dev/api/mcp` | The hosted [MCP](https://xdive.dev/docs/mcp.md) server (Streamable HTTP). | ## Choose the endpoint or tool Pick the cheapest call that answers the need. MCP tools call the same API, so the same limits and metering apply. | Job | REST endpoint | MCP tool | |---|---|---| | Find pages about a topic | `POST /orgs/{id}/search` | `search` | | Recent news | `POST /orgs/{id}/search` with `sources: ["news"]` and `freshness` | `search` with `sources: ["news"]` | | Read one known URL as Markdown | `POST /orgs/{id}/scrape` | `fetch` (or `scrape` for other formats) | | A page as HTML, links, a screenshot or structured JSON | `POST /orgs/{id}/scrape` with `formats` | `scrape` with `formats` | | A cited answer to a question | `POST /orgs/{id}/research` | `research` | | Fields or tables from up to 10 pages | `POST /orgs/{id}/extract` | `extract` | | Find the right URL on a site before reading it | `POST /orgs/{id}/map` with `search` | `map` | | Read many pages of one site | `POST /orgs/{id}/crawl`, then `GET /orgs/{id}/crawls/{crawlId}` | `crawl`, then `crawl_status` | | A known list of up to 100 URLs | `POST /orgs/{id}/batch/scrape` | (none; use `fetch` per URL) | | An `llms.txt` for someone else's site | `POST /orgs/{id}/llmstxt` | (none) | | Convert an uploaded PDF, DOCX or XLSX | `POST /orgs/{id}/parse` (see the OpenAPI spec) | (none) | | Click, type and scroll in a real browser | `POST /orgs/{id}/scrape` with [`actions`](https://xdive.dev/docs/scrape.md#browser-actions); interactive sessions under `/orgs/{id}/browser/sessions` (see the OpenAPI spec) | (none) | Rules of thumb: - Prefer search then scrape (or `fetch`) when you need to quote precisely; prefer research when the user wants an answer, not documents. - Do not crawl a site to read one page: map it with `search`, then scrape the page you need. - Keep crawls small and targeted with `max_pages`, `max_depth` and `include_paths`. - Crawl and batch scrape answer `202` with a job id. Poll `GET /orgs/{id}/crawls/{crawlId}`, honouring `Retry-After` and `ETag` ([details](https://xdive.dev/docs/crawl.md#polling-with-etag-and-retry-after)), or pass `webhook_url`. ## Authentication There are three ways for an agent to authenticate. See [Authentication](https://xdive.dev/docs/authentication.md) for the details. - **API key** (REST and MCP). Send `Authorization: Bearer ck_live_…` or `X-API-Key: ck_live_…`. A key belongs to one organization; call `GET /whoami` once to get its `orgId` and build `/orgs/{orgId}/…` paths from it. Keys carry scopes (`scrape`, `crawl`, `search`, `browser`, `connectors`); new keys get `scrape` and `crawl`, so add `search` for search and research. Keep keys in environment variables, never in prompts, code or logs. - **Device sign-in** (REST and MCP), for an agent or CLI without a key. Call `POST /api/oauth/device_authorization`, show the person the `verification_uri` and the `user_code`, and poll `POST /api/oauth/token` every `interval` seconds. Once the person approves at `/device`, the token response carries an API key and its `org_id`. See [Device sign-in](https://xdive.dev/docs/authentication.md#device-sign-in-for-clis-and-agents). - **OAuth** (MCP only). MCP clients such as Claude, ChatGPT, Cursor and Codex sign in with OAuth 2.1 and PKCE: paste the server URL, sign in, choose the organization, approve. No key is copied. OAuth tokens work only on `/api/mcp`, not on the REST API. API keys cannot manage the organization: API keys, usage, schedules, history and exports need a dashboard session. ## Costs and limits Usage is metered per organization and month in pages, searches and browser seconds. Map, status polling, cancellation and `GET /whoami` are free, and so are requests that are rejected before any work starts. See [Usage and billing](https://xdive.dev/docs/usage.md) for what each endpoint counts, and [Errors and limits](https://xdive.dev/docs/errors.md#quotas) for the monthly quotas. A used-up quota answers `402`; that is something to tell the user, not to retry. If a person gives you an API key, it may have its own monthly page and search budget, set so an agent cannot spend the whole organization's quota. Over that budget the API also answers `402` `QUOTA_EXCEEDED`, and the message names the key's budget. See [Per-key budgets](https://xdive.dev/docs/usage.md#per-key-budgets). Synchronous calls (scrape, map, search, research, extract, llms.txt, parse) time out on the server: about 30 seconds for a scrape, longer for deep research. Set client timeouts to at least 120 seconds, and 180 seconds for `depth: "deep"` research. ## Rate limits Each limit answers `429` (or `503` when a server is at capacity) with a `Retry-After` header in seconds. See [Rate limits and concurrency](https://xdive.dev/docs/errors.md#rate-limits-and-concurrency) for the numbers per plan. Work-creating requests also report the per-minute budget in standard headers: | Header | Meaning | |---|---| | `RateLimit-Limit` | Requests allowed in the current window. | | `RateLimit-Remaining` | Requests left in the current window. | | `RateLimit-Reset` | Seconds until the window resets. | The older `X-RateLimit-Limit` and `X-RateLimit-Remaining` headers carry the same numbers. Slow down when `RateLimit-Remaining` gets close to zero instead of waiting for a `429`. ## Retries and idempotency - **`429` and `503`:** wait `Retry-After` seconds (1 when it is missing), then retry. Cap the number of retries. - **`502` and `504`:** a dependency failed or was slow. One retry may succeed, and is metered like any request. - **`402`, `401`, `403` and other `4xx`:** do not retry the same request. Fix it or report it. - Every error body has `retryable`: retry only when it is `true`. Branch on the HTTP status and the error body's `code` (for example `CONCURRENCY_LIMIT`), never on the `error` text. Include the `X-Request-ID` response header when you report a problem. To retry a `POST` safely, send an `Idempotency-Key` header with a unique value (a UUID) per logical request, and reuse it on every retry of that request: ```bash curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \ -H "Authorization: Bearer $XDIVE_API_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: 7f0c2a9e-3b1d-4c55-9a7e-2d4f6b8c1e03" \ -d '{"url": "https://docs.example.com", "max_pages": 50}' ``` - The same key with the same body within 24 hours returns the stored response, with the header `Idempotent-Replayed: true`, instead of doing the work (and metering it) again. A retried crawl therefore does not start a second crawl. - The same key with a different body is rejected with `422` and the code `IDEMPOTENCY_KEY_REUSED`. Use a new key for a new request. ## Instructions for your agent Drop-in instruction files that teach a coding agent to use the API correctly: | File | For | Install | |---|---|---| | [AGENTS.md](https://xdive.dev/agents/AGENTS.md) | Codex, Cursor, Copilot, Gemini CLI and other agents that read `AGENTS.md` | Append it to the `AGENTS.md` at the root of your repository. | | [SKILL.md](https://xdive.dev/agents/xdive-api/SKILL.md) | Claude Code and the Claude Agent SDK | Save it as `.claude/skills/xdive-api/SKILL.md` in your repository (or `~/.claude/skills/xdive-api/SKILL.md`). | | [xdive.mdc](https://xdive.dev/agents/xdive.mdc) | Cursor | Save it as `.cursor/rules/xdive.mdc`. | ```bash curl -fsSL https://xdive.dev/agents/AGENTS.md >> AGENTS.md mkdir -p .claude/skills/xdive-api && curl -fsSL -o .claude/skills/xdive-api/SKILL.md https://xdive.dev/agents/xdive-api/SKILL.md mkdir -p .cursor/rules && curl -fsSL -o .cursor/rules/xdive.mdc https://xdive.dev/agents/xdive.mdc ``` For assistants that use the tools directly rather than writing code against the API, connect the [MCP server](https://xdive.dev/docs/mcp.md); its plugin for Claude Code and Codex adds a `web-research` skill on top of the same tools. --- Source: https://xdive.dev/docs/agents · All docs for agents: https://xdive.dev/llms.txt