## XDive web API

This project reads the web through the XDive API (scrape, crawl, map, search, research, extract).

- From a shell, prefer the CLI: `npx xdive scrape <url>` (Markdown on stdout), `npx xdive search "<query>"`, `npx xdive research "<question>"`. It reads `XDIVE_API_KEY`; without one, ask the user to run `npx xdive login`. Exit codes: 2 usage, 3 not signed in, 4 quota (do not retry), 5 rate limited.
- Docs for agents: https://xdive.dev/llms.txt (every page as Markdown); OpenAPI 3.1: https://xdive.dev/openapi.json. Check them instead of guessing parameters.
- Base URL `https://xdive.dev/api`. Auth: `Authorization: Bearer $XDIVE_API_KEY` (keys start with `ck_live_`). Read the key from the environment; never hard-code, log or commit it.
- Paths are `/orgs/{orgId}/...`. Get `orgId` once from `GET /whoami` and reuse it.
- Pick the cheapest call: `POST /orgs/{id}/scrape` for one URL (`{"url": ..., "formats": ["markdown"]}`; Markdown is at `data.artifacts.markdown.content`), `/search` to find pages, `/research` for a cited answer, `/extract` for structured fields from up to 10 URLs, `/map` to list a site's URLs, `/crawl` or `/batch/scrape` for many pages.
- `/crawl` and `/batch/scrape` return `202` with an id: poll `GET /orgs/{id}/crawls/{crawlId}` honouring `Retry-After`, or use `webhook_url`. Keep `max_pages` small.
- Errors are JSON `{"error": "...", "code": "..."}`. Branch on the status and `code`, not the message.
- Retry `429` and `503` after `Retry-After` seconds, with a cap. Do not retry `402` (quota used up) or other `4xx`.
- Send an `Idempotency-Key: <uuid>` header on POSTs you may retry, and reuse it on the retry. A replay has `Idempotent-Replayed: true`; a reused key with a different body fails with `422 IDEMPOTENCY_KEY_REUSED`.
- A page may carry `warnings` when its content looks incomplete; retry once with `"wait_for": 3000` or tell the user. For long pages send `"max_chars": 8000` and continue with `"offset"` set to the returned `next_offset`; `estimated_tokens` says what a part costs to read.
- Set HTTP timeouts to at least 120 s (180 s for `"depth": "deep"` research).