# XDive

> XDive is an HTTP API that turns web pages into data for code and AI agents: it scrapes one page as Markdown, HTML or structured JSON, crawls and maps whole sites, searches the web and news, extracts structured data with a prompt or JSON Schema, and answers questions with verified citations. It is available over REST (base URL https://xdive.dev/api, API key auth), through SDKs for Python, JavaScript, Go, Java, .NET, PHP, Ruby and Rust, and as a hosted MCP server at https://xdive.dev/api/mcp (OAuth or API key).

Every docs page below is plain Markdown. Paths in the API reference are relative to the base URL and start with `/orgs/{id}/`; `GET /whoami` with an API key returns the organization id. Errors are JSON with `error` and, for errors a program should handle, `code`.

## Get started

- [Introduction](https://xdive.dev/docs/introduction.md): What the API does, how it is organized, and where to start.
- [Quickstart](https://xdive.dev/docs/quickstart.md): Create an API key and make your first scrape, search and research call.
- [Authentication](https://xdive.dev/docs/authentication.md): API keys, scopes, organizations, sessions and OAuth for AI assistants.
- [For AI agents](https://xdive.dev/docs/agents.md): Which endpoint or MCP tool fits which job, auth, limits, safe retries, and Markdown docs, llms.txt and OpenAPI for agents.

## SDKs

- [Command line](https://xdive.dev/docs/cli.md): The xdive CLI for shells and coding agents: npx xdive scrape, search, research; device sign-in and exit codes.
- [Python SDK](https://xdive.dev/docs/sdk-python.md): Install, configure and use the synchronous and asynchronous Python clients.
- [JavaScript SDK](https://xdive.dev/docs/sdk-javascript.md): Install, configure and use the TypeScript client in Node.js and modern runtimes.
- [Go SDK](https://xdive.dev/docs/sdk-go.md): Install, configure and use the Go client with context, typed errors and safe retries.
- [Java SDK](https://xdive.dev/docs/sdk-java.md): Install and use the Java client (Java 11+, also Kotlin and Scala) with typed results and safe retries.
- [.NET SDK](https://xdive.dev/docs/sdk-dotnet.md): Install and use the C# client (NuGet) with async calls, dependency injection and safe retries.
- [PHP SDK](https://xdive.dev/docs/sdk-php.md): Install and use the PHP client with PSR-18, Laravel and Symfony examples.
- [Ruby SDK](https://xdive.dev/docs/sdk-ruby.md): Install and use the Ruby gem, with a Rails initializer and typed errors.
- [Rust SDK](https://xdive.dev/docs/sdk-rust.md): Install and use the async Rust crate with typed errors and safe retries.
- [Vite plugin](https://xdive.dev/docs/vite-plugin.md): Fetch pages as Markdown at build time and import them in a Vite app, with an optional llms.txt.

## API reference

- [Scrape](https://xdive.dev/docs/scrape.md): Fetch one page as Markdown, HTML, links, screenshots or structured data.
- [Crawl](https://xdive.dev/docs/crawl.md): Crawl a site asynchronously, follow its status, cancel it and receive webhooks.
- [Batch scrape](https://xdive.dev/docs/batch-scrape.md): Scrape up to 100 URLs as one job.
- [Map](https://xdive.dev/docs/map.md): Discover a site's URLs from its sitemaps and links without fetching content.
- [Search](https://xdive.dev/docs/search.md): Search the web and news, with relevant passages, scraping and a cited answer.
- [Research](https://xdive.dev/docs/research.md): Answer a question from the web with verified citations.
- [Extract](https://xdive.dev/docs/extract.md): Pull structured data from up to 10 URLs with a prompt or JSON Schema.
- [llms.txt](https://xdive.dev/docs/llms-txt.md): Generate llms.txt and llms-full.txt for a site.

## Integrations

- [MCP (Claude, ChatGPT, Cursor)](https://xdive.dev/docs/mcp.md): Connect AI assistants to the API through the Model Context Protocol.
- [LangChain](https://xdive.dev/docs/langchain.md): Agent tools and a document loader for LangChain and LangChain.js.
- [LlamaIndex](https://xdive.dev/docs/llamaindex.md): Load pages and whole sites into LlamaIndex documents.

## Reference

- [Errors and limits](https://xdive.dev/docs/errors.md): Status codes, error codes, rate limits, retries and quotas.
- [Usage and billing](https://xdive.dev/docs/usage.md): What is metered, what is free, and how to read usage.
- [Self-hosting](https://xdive.dev/docs/self-hosting.md): Run the platform on your own servers.
- [Changelog](https://xdive.dev/docs/changelog.md): What changed in the API and SDKs.

## Machine-readable

- [OpenAPI 3.1 spec](https://xdive.dev/openapi.json): every public REST endpoint, its parameters, responses and error codes.
- [MCP server](https://xdive.dev/api/mcp): Streamable HTTP endpoint with the tools search, fetch, research, scrape, map, crawl, crawl_status and extract; sign in with OAuth or send an API key.
- [AGENTS.md snippet](https://xdive.dev/agents/AGENTS.md): instructions to drop into a repository that calls the API.
- [Claude skill](https://xdive.dev/agents/xdive-api/SKILL.md): a SKILL.md for agents that call the REST API.
- [Cursor rule](https://xdive.dev/agents/xdive.mdc): the same guidance as a Cursor project rule.

## Optional

- [Full documentation](https://xdive.dev/llms-full.txt): every page above in one file.