# Introduction

XDive is an HTTP API that turns web pages into data your code and AI agents can use. It fetches a page as Markdown, HTML or structured JSON, crawls whole sites, searches the web, and answers questions with cited sources. Use it directly over HTTP or through the official SDKs for Python, JavaScript, Go, Java, .NET, PHP, Ruby and Rust.

## What the API does

| Capability | Endpoint | What it does |
|---|---|---|
| [Scrape](https://xdive.dev/docs/scrape.md) | `POST /orgs/{id}/scrape` | One page as Markdown, HTML, links, text, a screenshot or JSON, returned in the same request. |
| [Crawl](https://xdive.dev/docs/crawl.md) | `POST /orgs/{id}/crawl` | Follows a site's links in the background; poll for status or receive webhooks. |
| [Batch scrape](https://xdive.dev/docs/batch-scrape.md) | `POST /orgs/{id}/batch/scrape` | Up to 100 URLs as one background job. |
| [Map](https://xdive.dev/docs/map.md) | `POST /orgs/{id}/map` | A site's URLs from its sitemaps and links, without fetching content. |
| [Search](https://xdive.dev/docs/search.md) | `POST /orgs/{id}/search` | Web, news and image search, optionally with page content and a cited answer. |
| [Research](https://xdive.dev/docs/research.md) | `POST /orgs/{id}/research` | Answers a question from the web with verified citations. |
| [Extract](https://xdive.dev/docs/extract.md) | `POST /orgs/{id}/extract` | Structured data from up to 10 URLs, driven by a prompt or a JSON Schema. |
| [llms.txt](https://xdive.dev/docs/llms-txt.md) | `POST /orgs/{id}/llmstxt` | Generates `llms.txt` and `llms-full.txt` for a site. |
| [MCP](https://xdive.dev/docs/mcp.md) | `POST /mcp` | The same tools for AI assistants such as Claude, ChatGPT and Cursor. |

## Base URL

```text
https://xdive.dev/api
```

A self-hosted server uses its own origin, for example `https://your-host/api` (see [Self-hosting](https://xdive.dev/docs/self-hosting.md)). Requests and responses are JSON; send `Content-Type: application/json` with a request body. The API is described in OpenAPI 3.1 at [`/openapi.json`](https://xdive.dev/openapi.json), and these docs are available to AI agents as Markdown through [`/llms.txt`](https://xdive.dev/llms.txt) (see [For AI agents](https://xdive.dev/docs/agents.md)).

## How requests are organized

Every account belongs to one or more organizations, and usage, quotas and API keys belong to an organization. Endpoints that do work are scoped to one, so their paths start with `/orgs/{id}/`, where `{id}` looks like `org_8f2c41d09a7b4e6f93d0c5a1b2e47f18`.

An API key belongs to exactly one organization. `GET /whoami` tells you which one, so you only need to configure the key: the SDKs call `/whoami` once, cache the `orgId` and build every path from it. A key sent to another organization's path is rejected with `401`. See [Authentication](https://xdive.dev/docs/authentication.md).

Scrape, map, search, research, extract and llms.txt answer within the request. Crawl and batch scrape return `202 Accepted` with a job id at once and run on background workers.

## A first request

Create a key in the dashboard under **API Keys** (the [Quickstart](https://xdive.dev/docs/quickstart.md) walks through it), export it as `XDIVE_API_KEY`, then scrape a page:

**cURL**

```bash
# Look up the key's organization once (uses jq), then scrape.
ORG_ID=$(curl -s https://xdive.dev/api/whoami \
  -H "X-API-Key: $XDIVE_API_KEY" | jq -r .orgId)

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape("https://example.com")
print(page["artifacts"]["markdown"]["content"])
```

**JavaScript**

```ts
import { XDive, pageMarkdown } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com");
console.log(pageMarkdown(page));
```

## Pick your path

- **New to the API:** the [Quickstart](https://xdive.dev/docs/quickstart.md) goes from sign-up to a first scrape, search and research call.
- **Using an SDK:** [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md), [Go](https://xdive.dev/docs/sdk-go.md), [Java](https://xdive.dev/docs/sdk-java.md), [.NET](https://xdive.dev/docs/sdk-dotnet.md), [PHP](https://xdive.dev/docs/sdk-php.md), [Ruby](https://xdive.dev/docs/sdk-ruby.md) or [Rust](https://xdive.dev/docs/sdk-rust.md), and the [Vite plugin](https://xdive.dev/docs/vite-plugin.md).
- **One page or a whole site:** [Scrape](https://xdive.dev/docs/scrape.md), [Crawl](https://xdive.dev/docs/crawl.md), [Map](https://xdive.dev/docs/map.md) and [Batch scrape](https://xdive.dev/docs/batch-scrape.md).
- **Answers from the web:** [Search](https://xdive.dev/docs/search.md) and [Research](https://xdive.dev/docs/research.md).
- **Structured data:** [Extract](https://xdive.dev/docs/extract.md), or the `json` format on [Scrape](https://xdive.dev/docs/scrape.md).
- **AI assistants and agent frameworks:** [MCP](https://xdive.dev/docs/mcp.md), [LangChain](https://xdive.dev/docs/langchain.md) and [LlamaIndex](https://xdive.dev/docs/llamaindex.md).
- **Before going to production:** [Authentication](https://xdive.dev/docs/authentication.md), [Errors and limits](https://xdive.dev/docs/errors.md) and [Usage and billing](https://xdive.dev/docs/usage.md).
- **On your own servers:** [Self-hosting](https://xdive.dev/docs/self-hosting.md).

---

Source: https://xdive.dev/docs/introduction · All docs for agents: https://xdive.dev/llms.txt