# JavaScript SDK The `xdive` package is a TypeScript client for Node.js and other JavaScript runtimes. Each method makes one API call, or polls until a job finishes, and resolves to the parsed JSON with TypeScript types. The client discovers your organization from the API key, retries rate-limited requests, and throws typed errors. ## Install ```bash npm install xdive pnpm add xdive yarn add xdive bun add xdive ``` Runtime requirements: - **Node.js 18 or later**, or any runtime with a global `fetch`, `AbortController` and `setTimeout`. On a runtime without a global `fetch`, pass one in the [`fetch` option](#custom-fetch). - **ES modules.** The package is ESM only. From CommonJS, load it with `await import("xdive")`. - **TypeScript** declarations are included. To resolve the `xdive/langchain` entry point, use `"moduleResolution": "node16"`, `"nodenext"` or `"bundler"`. The package has no runtime dependencies. The [LangChain.js tools](https://xdive.dev/docs/langchain.md) need `@langchain/core` and `zod`, declared as optional peer dependencies. > **Warning:** > > An API key gives access to your organization's quota. Use the client on a server, not in code shipped to browsers. ## Configure the client ```ts import { XDive } from "xdive"; const xdive = new XDive(); // reads XDIVE_API_KEY and, if set, XDIVE_BASE_URL const configured = new XDive({ apiKey: process.env.XDIVE_API_KEY, baseUrl: "https://xdive.dev/api", orgId: "org_3f9a1c2e7b5d4a6f8e0c1b2a3d4e5f60", timeoutMs: 60_000, maxRetries: 5, }); ``` `new XDive(options?: XDiveOptions)` accepts: | Option | Type | Environment variable | Default | Description | |---|---|---|---|---| | `apiKey` | `string` | `XDIVE_API_KEY` | required | Organization API key (`ck_live_…`). The constructor throws when neither is set. | | `baseUrl` | `string` | `XDIVE_BASE_URL` | `https://xdive.dev/api` | API base URL, including `/api`. Trailing slashes are removed. | | `orgId` | `string` | none | discovered | Organization id. Pass it to skip the `/whoami` lookup. | | `timeoutMs` | `number` | none | `120000` | Per-request timeout in milliseconds. | | `maxRetries` | `number` | none | `3` | How many times an overload response (`429`, or `503` with code `SERVER_BUSY`) is retried. See [Retries](#retries-and-rate-limits). | | `fetch` | `typeof fetch` | none | `globalThis.fetch` | The `fetch` implementation to use. See [Custom fetch](#custom-fetch). | Environment variables are read from `process.env` when the runtime has one. Every request sends the key in the `X-API-Key` header. An API key belongs to one organization, and organization endpoints live under `/orgs/{id}/`. The first call that needs the organization id calls `GET /whoami` and caches the `orgId` for the life of the client; `await xdive.orgId()` returns it. The key's scopes decide which methods work: `scrape` for scraping, `crawl` for crawls and maps, `search` for search and research. A call outside the key's scopes rejects with a `XDiveError` with status `403`. See [Authentication](https://xdive.dev/docs/authentication.md). ## A first request ```ts import { XDive, pageMarkdown } from "xdive"; const xdive = new XDive(); const page = await xdive.scrape("https://example.com"); console.log(page.title); console.log(pageMarkdown(page)); ``` `scrape` resolves to the page object from the response's `data` field. Each requested format is an entry under `artifacts`; Markdown is the default format. `pageMarkdown(page)` returns the Markdown whether it is in `artifacts.markdown.content` (scrape, or a crawl with `formats`) or in `markdown` (a crawl without `formats`), and an empty string when there is none. ## Methods Every method returns a promise. Options are sent as the API's JSON fields, unchanged, so every field in the [API reference](https://xdive.dev/docs/scrape.md) is available even when the type does not name it. Options that are `undefined` or `null` are left out of the request. `formats` takes the API's format names (the `FormatName` type): `markdown`, `text`, `html`, `rawHtml`, `links`, `images`, `json`, `summary`, `question`, `highlights`, `branding`, `audio`, `screenshot`, `product` and `menu` (`audio` and `screenshot` on `scrape` only). A name is sent as `{ type: "markdown" }`; pass an object, such as `{ type: "json", schema: { … } }` or `{ type: "question", question: "…" }`, to set a format's options. | Method | API call | Resolves to | |---|---|---| | `orgId()` | `GET /whoami`, once | `string` | | `whoami()` | `GET /whoami` | `Json` | | `scrape(url, options?)` | `POST /orgs/{id}/scrape` | `Page` | | `crawl(url, options?)` | `POST /orgs/{id}/crawl`, then polls | `CrawlRecord` (finished) | | `startCrawl(url, options?)` | `POST /orgs/{id}/crawl` | `string` (crawl id) | | `getCrawl(crawlId)` | `GET /orgs/{id}/crawls/{crawlId}` | `CrawlRecord` | | `waitForCrawl(crawlId, options?)` | polls `GET /orgs/{id}/crawls/{crawlId}` | `CrawlRecord` (finished) | | `cancelCrawl(crawlId)` | `DELETE /orgs/{id}/crawls/{crawlId}` | `void` | | `crawlErrors(crawlId)` | `GET /orgs/{id}/crawls/{crawlId}/errors` | `CrawlErrors` | | `batchScrape(urls, options?)` | `POST /orgs/{id}/batch/scrape`, then polls | `CrawlRecord` (finished), or `string` (job id) with `wait: false` | | `map(url, options?)` | `POST /orgs/{id}/map` | `string[]` | | `search(query, options?)` | `POST /orgs/{id}/search` | `SearchResponse` | | `research(query, options?)` | `POST /orgs/{id}/research` | `ResearchResult` | | `extract(urls, options?)` | `POST /orgs/{id}/extract` | `Json` | | `llmsTxt(url, options?)` | `POST /orgs/{id}/llmstxt` | `Json` | ### whoami and orgId ```ts const me = await xdive.whoami(); // { auth: "api_key", keyId: "key_9b2e…", orgId: "org_3f9a…", scopes: ["crawl", "scrape", "search"], plan: "free", suspended: false } const orgId = await xdive.orgId(); ``` ### scrape ```ts const page = await xdive.scrape("https://example.com/pricing", { formats: ["markdown", "links"], max_age: 3_600_000, // accept a cached copy up to an hour old (milliseconds) }); ``` `scrape(url: string, options?: { formats?: Format[] } & Json): Promise`. Other options (`headers`, `respect_robots`, `redact_pii`, `actions`, …) are the [scrape parameters](https://xdive.dev/docs/scrape.md). ### crawl ```ts const record = await xdive.crawl("https://docs.example.com", { max_pages: 50, max_depth: 2 }); for (const page of record.results ?? []) { console.log(page.url, page.error ?? pageMarkdown(page).length); } ``` `crawl(url: string, options?: { formats?: Format[]; timeoutMs?: number; pollIntervalMs?: number } & Json): Promise` starts a crawl, then polls until it reaches `success`, `error` or `cancelled`. `timeoutMs` (default `600000`) bounds the wait, not the crawl: when it runs out the promise rejects with `CrawlTimeoutError` and the crawl keeps running on the server. `pollIntervalMs` (default `2000`) applies only when the server sends no `Retry-After`. A failed page stays in `results` with an `error` field. Crawl options (`max_pages`, `max_depth`, `include_paths`, `exclude_paths`, `webhook_url`, …) are the [crawl parameters](https://xdive.dev/docs/crawl.md). To start a crawl and come back to it later: ```ts const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 500 }); const { status } = await xdive.getCrawl(crawlId); // "queued" | "running" | "success" | "error" | "cancelled" const finished = await xdive.waitForCrawl(crawlId, { timeoutMs: 1_800_000, pollIntervalMs: 5000 }); ``` ### cancelCrawl ```ts await xdive.cancelCrawl(crawlId); ``` Cancels a queued or running crawl or batch scrape; the pages fetched so far are kept. A crawl that already finished rejects with a `XDiveError` with status `409`. ### crawlErrors ```ts const report = await xdive.crawlErrors(crawlId); for (const failure of report.errors) console.log(failure.url, failure.error); console.log("blocked by robots.txt:", report.robots_blocked); ``` Lists the pages of a crawl or batch scrape that failed, without downloading the results. ### batchScrape ```ts const record = await xdive.batchScrape(["https://example.com/a", "https://example.com/b"], { formats: ["markdown"], timeoutMs: 300_000, }); ``` `batchScrape(urls: string[], options?: { formats?: Format[]; timeoutMs?: number; wait?: boolean } & Json)` scrapes up to 100 URLs as one job. By default it waits for the job and resolves to the finished `CrawlRecord`. With `wait: false` it resolves to the job id as soon as the job is accepted; the overloads type that result as `string`: ```ts const jobId = await xdive.batchScrape(urls, { wait: false }); // string const record = await xdive.waitForCrawl(jobId); ``` The job is a crawl, so `getCrawl`, `waitForCrawl`, `cancelCrawl` and `crawlErrors` work on its id. See [Batch scrape](https://xdive.dev/docs/batch-scrape.md). ### map ```ts const urls = await xdive.map("https://example.com", { search: "pricing", limit: 100 }); ``` `map(url: string, options?: { search?: string; limit?: number; max_depth?: number }): Promise` returns the site's URLs, found from its sitemaps and links, without fetching their content. See [Map](https://xdive.dev/docs/map.md). ### search ```ts const response = await xdive.search("node.js test runner", { limit: 5, search_depth: "advanced" }); for (const result of response.results) { console.log(result.rank, result.url, result.relevance_score); for (const passage of result.passages ?? []) console.log(" ", passage.text.slice(0, 120)); } ``` `search(query: string, options?: Json): Promise`. With `search_depth: "advanced"`, the top results also carry `passages` (the excerpts most relevant to the query) and a `relevance_score`; results stay in the search engine's order. Other options (`sources`, `country`, `freshness`, `include_domains`, `auto_parameters`, `include_answer`, `scrape`, …) are the [search parameters](https://xdive.dev/docs/search.md). ### research ```ts const result = await xdive.research("What does PEP 639 change about license metadata?", { depth: "quick" }); console.log(result.answer); for (const citation of result.citations) console.log(citation.number, citation.url); ``` `research(query: string, options?: { depth?: "quick" | "deep"; output?: "answer" | "report"; schema?: Json; no_cache?: boolean } & Json): Promise`. `depth` defaults to `"quick"` (seconds); `"deep"` takes one to two minutes and plans sub-queries. Pass `schema` (a JSON Schema) to get structured `data` instead of prose, and `no_cache: true` to bypass the short-lived result cache (`cached` is `true` on a cache hit). `usage.plan`, when present, says how the question was planned: `"single"` (searched directly), `"multi_hop"` (answered through an intermediate search) or `"unavailable"` (the planning step was too slow or failed). See [Research](https://xdive.dev/docs/research.md). ### extract ```ts const result = await xdive.extract(["https://example.com/pricing", "https://example.com/plans"], { prompt: "List each plan with its monthly price", }); ``` `extract(urls: string[], options?: { prompt?: string; schema?: Json } & Json): Promise` extracts structured data from up to 10 URLs. See [Extract](https://xdive.dev/docs/extract.md). ### llmsTxt ```ts const files = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true }); console.log(files.llms_txt); ``` `llmsTxt(url: string, options?: { max_urls?: number; full_text?: boolean }): Promise` generates an llms.txt index, and with `full_text` also `llms_full_txt`. See [llms.txt](https://xdive.dev/docs/llms-txt.md). ## Types The package exports these types. Response types list the fields the SDK relies on and allow any other field the API returns. | Type | Describes | |---|---| | `XDiveOptions` | The constructor options above. | | `Json` | `Record`, used for options and untyped responses. | | `FormatName` | The format names listed under [Methods](#methods); `"raw_html"` is accepted as an alias of `"rawHtml"`. | | `Format` | A `FormatName`, or an object `{ type: FormatName, prompt?, schema?, question?, query?, max_results? }` that also sets the format's options. | | `Page` | A scraped page: `url`, `title`, `description`, `markdown`, `artifacts`, `error`. | | `CrawlRecord` | A crawl or batch scrape: `id`, `status`, `pages`, `error`, `results` (`Page[]`). | | `CrawlErrors` | `id`, `status`, `error`, `errors` (`url` and `error` pairs), `robots_blocked` (URLs). | | `SearchResponse` | `query`, `provider`, `results`, and `answer` and `parameters` when requested. | | `SearchResult` | `id`, `source`, `rank`, `title`, `url`, `description`, `published_at`, `page`, `passages`, `relevance_score`. | | `Passage` | `text`, `score`, `start` (offset in the source text). | | `ResearchResult` | `query`, `answer` or `data`, `citations`, `sources`, `queries`, `confidence`, `usage` (with `plan`), `cached`. | | `Citation` | `number`, `url`, `title`, `quote`. | ```ts import type { CrawlRecord, Page, ResearchResult } from "xdive"; ``` The constant `DEFAULT_BASE_URL` holds the hosted API base URL. ## Errors All error classes extend `XDiveError`, which extends `Error`. | Class | Thrown when | Properties | |---|---|---| | `XDiveError` | The API answers with status `400` or above. | `status` (number), `code` (string or `undefined`), `message` | | `RateLimitedError` | Still overloaded (`429`, or `503` with code `SERVER_BUSY`) after `maxRetries` retries. | `retryAfterSeconds` (number), plus the above | | `QuotaExceededError` | The API answers `402`: the organization's monthly quota is used up. | as `XDiveError` | | `CrawlTimeoutError` | `crawl`, `waitForCrawl` or `batchScrape` waited longer than `timeoutMs`. `status` is `408`; no HTTP request failed. | `crawlId` | `code` is the API's machine-readable code when it sends one, for example `CONCURRENCY_LIMIT` or `SERVER_BUSY`. `message` combines the status, the code and the API's `error` text: `429 CONCURRENCY_LIMIT: too many concurrent requests…`, or `402: …` without a code. ```ts import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive"; const xdive = new XDive(); try { const page = await xdive.scrape("https://example.com"); } catch (error) { if (error instanceof RateLimitedError) console.log(`still throttled, retry in ${error.retryAfterSeconds}s`); else if (error instanceof QuotaExceededError) console.log("monthly quota used up"); else if (error instanceof XDiveError) console.log(error.status, error.code, error.message); else throw error; // network failure or timeout } ``` Network failures and timeouts are not wrapped: the promise rejects with whatever `fetch` threw. A request that exceeds `timeoutMs` is aborted, which rejects with an `AbortError`. See [Errors and limits](https://xdive.dev/docs/errors.md) for the API's status codes. ## Retries and rate limits The client retries only overload, the API's way of saying "slow down": - `429`: the organization exceeded its plan's request rate or concurrency. - `503` with the code `SERVER_BUSY` in the JSON body: the server is at capacity. Both carry a `Retry-After` header. The client waits that many seconds (1 second when the header is missing) and sends the request again, up to `maxRetries` times, so a request is attempted at most `maxRetries + 1` times; each attempt gets its own `timeoutMs`. When the last attempt is still throttled the promise rejects with `RateLimitedError`, whose `retryAfterSeconds` is the final `Retry-After`. Every other error rejects at once, including any other `503`, such as a feature the server has not configured or a job queue that is unavailable: you get a plain `XDiveError` with the API's `status`, `message` and `code`. `maxRetries: 0` turns retries off. `maxRetries` is a public property, so you can change it on an existing client. ## Crawl polling `crawl`, `waitForCrawl` and `batchScrape` poll `GET /orgs/{id}/crawls/{crawlId}`: - Each poll sends the `ETag` of the last response in `If-None-Match`. While nothing changed, the server answers `304` without a body. - Between polls the client waits the number of seconds in the server's `Retry-After` header, which is `2` while a crawl runs, and falls back to `pollIntervalMs` when the header is absent. The wait never runs past the deadline. - Polling stops at `success`, `error` or `cancelled` and resolves to that record. A crawl that ends in `error` resolves; check `record.status`. - After `timeoutMs` it rejects with `CrawlTimeoutError`. The crawl itself is not cancelled. ## Custom fetch Pass `fetch` to log or instrument requests, to route them through a proxy, or to provide `fetch` on a runtime that lacks a global one. The client calls it with a URL string and a `RequestInit`, and expects a standard `Response`. ```ts import { XDive } from "xdive"; const xdive = new XDive({ fetch: async (input, init) => { const started = Date.now(); const response = await fetch(input, init); console.log(init?.method, String(input), response.status, `${Date.now() - started}ms`); return response; }, }); ``` Tests can pass a function that returns canned `Response` objects instead of calling the network. ## Self-hosted Point the client at your own server with `baseUrl`, including the `/api` prefix, or set `XDIVE_BASE_URL`: ```ts const xdive = new XDive({ apiKey: process.env.XDIVE_API_KEY, baseUrl: "https://your-host/api" }); ``` Keys are issued by the server they are used against. See [Self-hosting](https://xdive.dev/docs/self-hosting.md). ## Related - [Python SDK](https://xdive.dev/docs/sdk-python.md) - [LangChain](https://xdive.dev/docs/langchain.md) - [Errors and limits](https://xdive.dev/docs/errors.md) --- Source: https://xdive.dev/docs/sdk-javascript · All docs for agents: https://xdive.dev/llms.txt