Introduction

XDive is an HTTP API that turns web pages into data your code and AI agents can use. It fetches a page as Markdown, HTML or structured JSON, crawls whole sites, searches the web, and answers questions with cited sources. Use it directly over HTTP or through the official SDKs for Python, JavaScript, Go, Java, .NET, PHP, Ruby and Rust.

What the API does

Capability	Endpoint	What it does
Scrape	`POST /orgs/{id}/scrape`	One page as Markdown, HTML, links, text, a screenshot or JSON, returned in the same request.
Crawl	`POST /orgs/{id}/crawl`	Follows a site's links in the background; poll for status or receive webhooks.
Batch scrape	`POST /orgs/{id}/batch/scrape`	Up to 100 URLs as one background job.
Map	`POST /orgs/{id}/map`	A site's URLs from its sitemaps and links, without fetching content.
Search	`POST /orgs/{id}/search`	Web, news and image search, optionally with page content and a cited answer.
Research	`POST /orgs/{id}/research`	Answers a question from the web with verified citations.
Extract	`POST /orgs/{id}/extract`	Structured data from up to 10 URLs, driven by a prompt or a JSON Schema.
llms.txt	`POST /orgs/{id}/llmstxt`	Generates `llms.txt` and `llms-full.txt` for a site.
MCP	`POST /mcp`	The same tools for AI assistants such as Claude, ChatGPT and Cursor.

Base URL

text

https://xdive.dev/api

A self-hosted server uses its own origin, for example https://your-host/api (see Self-hosting). Requests and responses are JSON; send Content-Type: application/json with a request body. The API is described in OpenAPI 3.1 at /openapi.json, and these docs are available to AI agents as Markdown through /llms.txt (see For AI agents).

How requests are organized

Every account belongs to one or more organizations, and usage, quotas and API keys belong to an organization. Endpoints that do work are scoped to one, so their paths start with /orgs/{id}/, where {id} looks like org_8f2c41d09a7b4e6f93d0c5a1b2e47f18.

An API key belongs to exactly one organization. GET /whoami tells you which one, so you only need to configure the key: the SDKs call /whoami once, cache the orgId and build every path from it. A key sent to another organization's path is rejected with 401. See Authentication.

Scrape, map, search, research, extract and llms.txt answer within the request. Crawl and batch scrape return 202 Accepted with a job id at once and run on background workers.

A first request

Create a key in the dashboard under API Keys (the Quickstart walks through it), export it as XDIVE_API_KEY, then scrape a page:

bash

# Look up the key's organization once (uses jq), then scrape.
ORG_ID=$(curl -s https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY" | jq -r .orgId)
 
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'

python

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape("https://example.com")
print(page["artifacts"]["markdown"]["content"])

import { XDive, pageMarkdown } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com");
console.log(pageMarkdown(page));

Pick your path

New to the API: the Quickstart goes from sign-up to a first scrape, search and research call.
Using an SDK: Python, JavaScript, Go, Java, .NET, PHP, Ruby or Rust, and the Vite plugin.
One page or a whole site: Scrape, Crawl, Map and Batch scrape.
Answers from the web: Search and Research.
Structured data: Extract, or the json format on Scrape.
AI assistants and agent frameworks: MCP, LangChain and LlamaIndex.
Before going to production: Authentication, Errors and limits and Usage and billing.
On your own servers: Self-hosting.