JavaScript SDK

The xdive package is a TypeScript client for Node.js and other JavaScript runtimes. Each method makes one API call, or polls until a job finishes, and resolves to the parsed JSON with TypeScript types. The client discovers your organization from the API key, retries rate-limited requests, and throws typed errors.

Install

npm install xdive
pnpm add xdive
yarn add xdive
bun add xdive

Runtime requirements:

• Node.js 18 or later, or any runtime with a global fetch, AbortController and setTimeout. On a runtime without a global fetch, pass one in the fetch option.
• ES modules. The package is ESM only. From CommonJS, load it with await import("xdive").
• TypeScript declarations are included. To resolve the xdive/langchain entry point, use "moduleResolution": "node16", "nodenext" or "bundler".

The package has no runtime dependencies. The LangChain.js tools need @langchain/core and zod, declared as optional peer dependencies.

Configure the client

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY and, if set, XDIVE_BASE_URL
 
const configured = new XDive({
  apiKey: process.env.XDIVE_API_KEY,
  baseUrl: "https://xdive.dev/api",
  orgId: "org_3f9a1c2e7b5d4a6f8e0c1b2a3d4e5f60",
  timeoutMs: 60_000,
  maxRetries: 5,
});

new XDive(options?: XDiveOptions) accepts:

Environment variables are read from process.env when the runtime has one. Every request sends the key in the X-API-Key header.

An API key belongs to one organization, and organization endpoints live under /orgs/{id}/. The first call that needs the organization id calls GET /whoami and caches the orgId for the life of the client; await xdive.orgId() returns it.

The key's scopes decide which methods work: scrape for scraping, crawl for crawls and maps, search for search and research. A call outside the key's scopes rejects with a XDiveError with status 403. See Authentication.

A first request

import { XDive, pageMarkdown } from "xdive";
 
const xdive = new XDive();
const page = await xdive.scrape("https://example.com");
console.log(page.title);
console.log(pageMarkdown(page));

scrape resolves to the page object from the response's data field. Each requested format is an entry under artifacts; Markdown is the default format. pageMarkdown(page) returns the Markdown whether it is in artifacts.markdown.content (scrape, or a crawl with formats) or in markdown (a crawl without formats), and an empty string when there is none.

Methods

Every method returns a promise. Options are sent as the API's JSON fields, unchanged, so every field in the API reference is available even when the type does not name it. Options that are undefined or null are left out of the request.

formats takes the API's format names (the FormatName type): markdown, text, html, rawHtml, links, images, json, summary, question, highlights, branding, audio, screenshot, product and menu (audio and screenshot on scrape only). A name is sent as { type: "markdown" }; pass an object, such as { type: "json", schema: { … } } or { type: "question", question: "…" }, to set a format's options.

whoami and orgId

const me = await xdive.whoami();
// { auth: "api_key", keyId: "key_9b2e…", orgId: "org_3f9a…", scopes: ["crawl", "scrape", "search"], plan: "free", suspended: false }
const orgId = await xdive.orgId();

scrape

const page = await xdive.scrape("https://example.com/pricing", {
  formats: ["markdown", "links"],
  max_age: 3_600_000, // accept a cached copy up to an hour old (milliseconds)
});

scrape(url: string, options?: { formats?: Format[] } & Json): Promise<Page>. Other options (headers, respect_robots, redact_pii, actions, …) are the scrape parameters.

crawl

const record = await xdive.crawl("https://docs.example.com", { max_pages: 50, max_depth: 2 });
for (const page of record.results ?? []) {
  console.log(page.url, page.error ?? pageMarkdown(page).length);
}

crawl(url: string, options?: { formats?: Format[]; timeoutMs?: number; pollIntervalMs?: number } & Json): Promise<CrawlRecord> starts a crawl, then polls until it reaches success, error or cancelled. timeoutMs (default 600000) bounds the wait, not the crawl: when it runs out the promise rejects with CrawlTimeoutError and the crawl keeps running on the server. pollIntervalMs (default 2000) applies only when the server sends no Retry-After. A failed page stays in results with an error field. Crawl options (max_pages, max_depth, include_paths, exclude_paths, webhook_url, …) are the crawl parameters.

To start a crawl and come back to it later:

const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 500 });
const { status } = await xdive.getCrawl(crawlId); // "queued" | "running" | "success" | "error" | "cancelled"
const finished = await xdive.waitForCrawl(crawlId, { timeoutMs: 1_800_000, pollIntervalMs: 5000 });

cancelCrawl

await xdive.cancelCrawl(crawlId);

Cancels a queued or running crawl or batch scrape; the pages fetched so far are kept. A crawl that already finished rejects with a XDiveError with status 409.

crawlErrors

const report = await xdive.crawlErrors(crawlId);
for (const failure of report.errors) console.log(failure.url, failure.error);
console.log("blocked by robots.txt:", report.robots_blocked);

Lists the pages of a crawl or batch scrape that failed, without downloading the results.

batchScrape

const record = await xdive.batchScrape(["https://example.com/a", "https://example.com/b"], {
  formats: ["markdown"],
  timeoutMs: 300_000,
});

batchScrape(urls: string[], options?: { formats?: Format[]; timeoutMs?: number; wait?: boolean } & Json) scrapes up to 100 URLs as one job. By default it waits for the job and resolves to the finished CrawlRecord. With wait: false it resolves to the job id as soon as the job is accepted; the overloads type that result as string:

const jobId = await xdive.batchScrape(urls, { wait: false }); // string
const record = await xdive.waitForCrawl(jobId);

The job is a crawl, so getCrawl, waitForCrawl, cancelCrawl and crawlErrors work on its id. See Batch scrape.

map

const urls = await xdive.map("https://example.com", { search: "pricing", limit: 100 });

map(url: string, options?: { search?: string; limit?: number; max_depth?: number }): Promise<string[]> returns the site's URLs, found from its sitemaps and links, without fetching their content. See Map.

search

const response = await xdive.search("node.js test runner", { limit: 5, search_depth: "advanced" });
for (const result of response.results) {
  console.log(result.rank, result.url, result.relevance_score);
  for (const passage of result.passages ?? []) console.log("  ", passage.text.slice(0, 120));
}

search(query: string, options?: Json): Promise<SearchResponse>. With search_depth: "advanced", the top results also carry passages (the excerpts most relevant to the query) and a relevance_score; results stay in the search engine's order. Other options (sources, country, freshness, include_domains, auto_parameters, include_answer, scrape, …) are the search parameters.

research

const result = await xdive.research("What does PEP 639 change about license metadata?", { depth: "quick" });
console.log(result.answer);
for (const citation of result.citations) console.log(citation.number, citation.url);

research(query: string, options?: { depth?: "quick" | "deep"; output?: "answer" | "report"; schema?: Json; no_cache?: boolean } & Json): Promise<ResearchResult>. depth defaults to "quick" (seconds); "deep" takes one to two minutes and plans sub-queries. Pass schema (a JSON Schema) to get structured data instead of prose, and no_cache: true to bypass the short-lived result cache (cached is true on a cache hit). usage.plan, when present, says how the question was planned: "single" (searched directly), "multi_hop" (answered through an intermediate search) or "unavailable" (the planning step was too slow or failed). See Research.

extract

const result = await xdive.extract(["https://example.com/pricing", "https://example.com/plans"], {
  prompt: "List each plan with its monthly price",
});

extract(urls: string[], options?: { prompt?: string; schema?: Json } & Json): Promise<Json> extracts structured data from up to 10 URLs. See Extract.

llmsTxt

const files = await xdive.llmsTxt("https://docs.example.com", { max_urls: 20, full_text: true });
console.log(files.llms_txt);

llmsTxt(url: string, options?: { max_urls?: number; full_text?: boolean }): Promise<Json> generates an llms.txt index, and with full_text also llms_full_txt. See llms.txt.

Types

The package exports these types. Response types list the fields the SDK relies on and allow any other field the API returns.

import type { CrawlRecord, Page, ResearchResult } from "xdive";

The constant DEFAULT_BASE_URL holds the hosted API base URL.

Errors

All error classes extend XDiveError, which extends Error.

code is the API's machine-readable code when it sends one, for example CONCURRENCY_LIMIT or SERVER_BUSY. message combines the status, the code and the API's error text: 429 CONCURRENCY_LIMIT: too many concurrent requests…, or 402: … without a code.

import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive";
 
const xdive = new XDive();
try {
  const page = await xdive.scrape("https://example.com");
} catch (error) {
  if (error instanceof RateLimitedError) console.log(`still throttled, retry in ${error.retryAfterSeconds}s`);
  else if (error instanceof QuotaExceededError) console.log("monthly quota used up");
  else if (error instanceof XDiveError) console.log(error.status, error.code, error.message);
  else throw error; // network failure or timeout
}

Network failures and timeouts are not wrapped: the promise rejects with whatever fetch threw. A request that exceeds timeoutMs is aborted, which rejects with an AbortError. See Errors and limits for the API's status codes.

Retries and rate limits

The client retries only overload, the API's way of saying "slow down":

• 429: the organization exceeded its plan's request rate or concurrency.
• 503 with the code SERVER_BUSY in the JSON body: the server is at capacity.

Both carry a Retry-After header. The client waits that many seconds (1 second when the header is missing) and sends the request again, up to maxRetries times, so a request is attempted at most maxRetries + 1 times; each attempt gets its own timeoutMs. When the last attempt is still throttled the promise rejects with RateLimitedError, whose retryAfterSeconds is the final Retry-After.

Every other error rejects at once, including any other 503, such as a feature the server has not configured or a job queue that is unavailable: you get a plain XDiveError with the API's status, message and code. maxRetries: 0 turns retries off. maxRetries is a public property, so you can change it on an existing client.

Crawl polling

crawl, waitForCrawl and batchScrape poll GET /orgs/{id}/crawls/{crawlId}:

• Each poll sends the ETag of the last response in If-None-Match. While nothing changed, the server answers 304 without a body.
• Between polls the client waits the number of seconds in the server's Retry-After header, which is 2 while a crawl runs, and falls back to pollIntervalMs when the header is absent. The wait never runs past the deadline.
• Polling stops at success, error or cancelled and resolves to that record. A crawl that ends in error resolves; check record.status.
• After timeoutMs it rejects with CrawlTimeoutError. The crawl itself is not cancelled.

Custom fetch

Pass fetch to log or instrument requests, to route them through a proxy, or to provide fetch on a runtime that lacks a global one. The client calls it with a URL string and a RequestInit, and expects a standard Response.

import { XDive } from "xdive";
 
const xdive = new XDive({
  fetch: async (input, init) => {
    const started = Date.now();
    const response = await fetch(input, init);
    console.log(init?.method, String(input), response.status, `${Date.now() - started}ms`);
    return response;
  },
});

Tests can pass a function that returns canned Response objects instead of calling the network.

Self-hosted

Point the client at your own server with baseUrl, including the /api prefix, or set XDIVE_BASE_URL:

const xdive = new XDive({ apiKey: process.env.XDIVE_API_KEY, baseUrl: "https://your-host/api" });

Keys are issued by the server they are used against. See Self-hosting.

Related

• Python SDK
• LangChain
• Errors and limits