# Changelog

User-facing changes to the API, the SDKs and the dashboard, newest first. Releases with only internal changes (v1.0.1 and v1.0.2) are left out.

## Unreleased

- SDKs: the packages are now called `xdive`. Install the Python package with `pip install xdive` (import `xdive`, classes `XDive` and `AsyncXDive`) and the JavaScript package with `npm install xdive`, `pnpm add xdive`, `yarn add xdive` or `bun add xdive` (class `XDive`). They read `XDIVE_API_KEY` and `XDIVE_BASE_URL`; the older `CRAWLER_API_KEY` and `CRAWLER_BASE_URL` still work.
- SDKs: every `POST` carries an `Idempotency-Key`, reused by each retry, and timeouts and dropped connections are retried safely.
- New SDKs for [Java](https://xdive.dev/docs/sdk-java.md) (`dev.xdive:xdive`), [.NET](https://xdive.dev/docs/sdk-dotnet.md) (`XDive` on NuGet), [PHP](https://xdive.dev/docs/sdk-php.md) (`xdive/xdive`), [Ruby](https://xdive.dev/docs/sdk-ruby.md) (`xdive`) and [Rust](https://xdive.dev/docs/sdk-rust.md) (`xdive`), with the same methods, typed errors, safe retries and an `Idempotency-Key` on every POST.
- New [Go SDK](https://xdive.dev/docs/sdk-go.md) (`github.com/xdive/xdive-go`) and [Vite plugin](https://xdive.dev/docs/vite-plugin.md) (`vite-plugin-xdive`), which fetches pages as Markdown at build time and can emit an `llms.txt`.
- Docs for AI agents: [`/llms.txt`](https://xdive.dev/llms.txt), [`/llms-full.txt`](https://xdive.dev/llms-full.txt), every docs page as Markdown at `/docs/<slug>.md`, an OpenAPI 3.1 spec at [`/openapi.json`](https://xdive.dev/openapi.json), drop-in `AGENTS.md`, Claude skill and Cursor rule files, and the new [For AI agents](https://xdive.dev/docs/agents.md) page.
- Research: identical requests from the same organization are served from a 10-minute result cache, marked `"cached": true` and not metered; `"no_cache": true` forces a new run.
- Research: quick mode answers once enough sources have arrived instead of waiting for the slowest page, and runs one follow-up search when its sources do not cover the question.
- Research: two-step questions are answered by first resolving the intermediate entity; `usage.hops` and `usage.plan` report what was done.
- Research: sources that could not be read in time contribute their search snippet, marked `"snippet": true`.
- Research: `usage` reports where the time went (`search_ms`, `read_ms`, `model_ms`), and a run no longer returns an empty answer.
- Research: `schema` must be a JSON object.
- Search: `search_depth: "advanced"` keeps the engines' ranking and adds each result's passages and `relevance_score` instead of re-ordering results.
- Search: `include_answer` has its own time budget, and a page read for both advanced search and scraping is counted once.
- Scrape: a site that the egress proxy provider refuses (typically government and banking sites) returns `403` with code `SITE_RESTRICTED` and a plain explanation; crawl pages carry `error_code: "SITE_RESTRICTED"`.
- Self-hosting: an optional egress proxy (`EGRESS_PROXY_URL`, `EGRESS_PROXY_DOMAINS`) for sites that refuse data-centre addresses; proxied pages report `fetch_route: "proxy"`.
- Security: fetches through the egress proxy are checked by the SSRF guard.
- Self-hosting: `REGISTRATION_APPROVAL=required` holds new sign-ups until a super admin approves them; sign-up answers `202` and sign-in returns `ACCOUNT_PENDING_APPROVAL` or `ACCOUNT_REJECTED`.
- Accounts: rejecting an account ends its sessions and OAuth grants at once and suspends the organizations it alone belongs to.
- Accounts: addresses in `SUPER_ADMIN_EMAILS` become super admins only after they are verified; sign-in returns `EMAIL_NOT_VERIFIED` until then.
- Self-hosting: `LLM_MODEL_ROTATION=round_robin` spreads requests across the configured models.
- Dashboard: fixed header and sidebar, and loading placeholders in place of spinners.
- SDKs: the research result types include `cached`, `hops`, snippet sources and phase timings.

## v1.5.0 (2026-10-10)

- New `POST /orgs/{id}/research`: answers (quick) and reports (deep) from the web with verified citations, or structured `data` from a JSON Schema. Also available as an MCP tool.
- Search: several engines (Brave and Serper) are queried together and their rankings merged, results are cached, and near-duplicates are removed.
- Search: `search_depth: "advanced"` returns each result's relevant passages and a `relevance_score`; `auto_parameters` infers sources and freshness from the query; `include_answer` adds a short cited answer.
- New Python SDK (`xdive`, with `XDive` and `AsyncXDive`) and JavaScript SDK (`xdive`), with LangChain tools and a LlamaIndex reader.
- New `GET /whoami`: the organization, scopes and plan of an API key, used by the SDKs to find the organization.
- New public `GET /status` summary for status pages and monitors.
- Language-model calls retry transient failures and fall back to `LLM_FALLBACK_MODELS`; a failing search engine is skipped.

## v1.4.0 (2026-10-09)

- Crawl status responses carry an `ETag`; sending it in `If-None-Match` returns `304` while nothing has changed. Running crawls suggest a polling interval with `Retry-After: 2`.
- Synchronous endpoints enforce a per-organization concurrency limit (`429 CONCURRENCY_LIMIT`) and per-server capacity (`503 SERVER_BUSY`), both with `Retry-After`.
- Crawl workers are shared fairly between organizations by plan, and browser-rendered crawls run in their own queue.
- Synchronous scrape is now checked against the page quota and counts one page per successful scrape.
- `max_age` can accept older cached content as well as require fresher content; simultaneous requests for the same uncached page share one fetch.
- Security: the page cache is keyed by request headers as well as URL, so a page fetched with cookies or authorization headers is no longer served to requests sent without them.
- With billing configured, `PUT /orgs/{id}/plan` only downgrades to `free`; paid plans are activated through checkout.

## v1.3.1 (2026-10-09)

- Security: headless Chrome traffic passes through the SSRF guard.
- Unreachable hosts fail fast with a clear error instead of escalating to the browser, and the scrape `502` message includes the page's error.

## v1.3.0 (2026-10-08)

- MCP clients such as ChatGPT, claude.ai, Claude Code and Codex connect with OAuth 2.1. Connected clients are listed, and can be revoked, under **API Keys** → **Connected apps**.
- Breaking: the MCP route with an API key in the URL (`/api/mcp/{key}`) is removed. Clients that send headers keep using API keys.

## v1.2.1 (2026-10-08)

- Fixed: `POST /api/mcp` no longer answers with a `301` redirect, which broke header-authenticated MCP clients.

## v1.2.0 (2026-10-07)

- MCP connectors for ChatGPT and claude.ai, and a new `fetch` tool that returns a page's full Markdown.
- `GET` and `DELETE` on the MCP endpoint return `405`.
- Redesigned landing and sign-in pages with light and dark themes.

## v1.1.1 (2026-10-07)

- Every default fetch path escalates past WAF bot filters with Chrome TLS impersonation, so protected sites return content instead of `403` or `404`. `adaptive_fetch: false` still sends a single plain request.

## v1.1.0 (2026-10-07)

- A public landing page for signed-out visitors.

## v1.0.3 (2026-10-07)

- Per-organization requests-per-minute budgets by plan (60, 600 and 6,000) on work-creating endpoints, reported in `X-RateLimit-Limit` and `X-RateLimit-Remaining`. Self-hosted servers can turn them off with `ORG_RATE_LIMITS=off`.

## v1.0.0 (2026-10-07)

First release.

- Scrape in many formats (Markdown, HTML, links, text, screenshots, JSON and more), crawl with webhooks, batch scrape, map, extract, llms.txt, document parsing, and web, news and image search.
- Adaptive fetching (plain HTTP, TLS impersonation, headless browser), interactive browser sessions, PII redaction, and zero-data-retention and lockdown modes.
- Organizations, API keys with scopes, expiry and IP allowlists, usage metering with plan quotas, Stripe billing and an audit log.
- Schedules with change monitoring, an MCP server (stdio and Streamable HTTP), the dashboard, and self-hosting with Docker Compose.

---

Source: https://xdive.dev/docs/changelog · All docs for agents: https://xdive.dev/llms.txt