XDiveDocs
Sign up
View .md

Changelog

User-facing changes to the API, the SDKs and the dashboard, newest first. Releases with only internal changes (v1.0.1 and v1.0.2) are left out.

Unreleased

  • SDKs: the packages are now called xdive. Install the Python package with pip install xdive (import xdive, classes XDive and AsyncXDive) and the JavaScript package with npm install xdive, pnpm add xdive, yarn add xdive or bun add xdive (class XDive). They read XDIVE_API_KEY and XDIVE_BASE_URL; the older CRAWLER_API_KEY and CRAWLER_BASE_URL still work.
  • SDKs: every POST carries an Idempotency-Key, reused by each retry, and timeouts and dropped connections are retried safely.
  • New SDKs for Java (dev.xdive:xdive), .NET (XDive on NuGet), PHP (xdive/xdive), Ruby (xdive) and Rust (xdive), with the same methods, typed errors, safe retries and an Idempotency-Key on every POST.
  • New Go SDK (github.com/xdive/xdive-go) and Vite plugin (vite-plugin-xdive), which fetches pages as Markdown at build time and can emit an llms.txt.
  • Docs for AI agents: /llms.txt, /llms-full.txt, every docs page as Markdown at /docs/<slug>.md, an OpenAPI 3.1 spec at /openapi.json, drop-in AGENTS.md, Claude skill and Cursor rule files, and the new For AI agents page.
  • Research: identical requests from the same organization are served from a 10-minute result cache, marked "cached": true and not metered; "no_cache": true forces a new run.
  • Research: quick mode answers once enough sources have arrived instead of waiting for the slowest page, and runs one follow-up search when its sources do not cover the question.
  • Research: two-step questions are answered by first resolving the intermediate entity; usage.hops and usage.plan report what was done.
  • Research: sources that could not be read in time contribute their search snippet, marked "snippet": true.
  • Research: usage reports where the time went (search_ms, read_ms, model_ms), and a run no longer returns an empty answer.
  • Research: schema must be a JSON object.
  • Search: search_depth: "advanced" keeps the engines' ranking and adds each result's passages and relevance_score instead of re-ordering results.
  • Search: include_answer has its own time budget, and a page read for both advanced search and scraping is counted once.
  • Scrape: a site that the egress proxy provider refuses (typically government and banking sites) returns 403 with code SITE_RESTRICTED and a plain explanation; crawl pages carry error_code: "SITE_RESTRICTED".
  • Self-hosting: an optional egress proxy (EGRESS_PROXY_URL, EGRESS_PROXY_DOMAINS) for sites that refuse data-centre addresses; proxied pages report fetch_route: "proxy".
  • Security: fetches through the egress proxy are checked by the SSRF guard.
  • Self-hosting: REGISTRATION_APPROVAL=required holds new sign-ups until a super admin approves them; sign-up answers 202 and sign-in returns ACCOUNT_PENDING_APPROVAL or ACCOUNT_REJECTED.
  • Accounts: rejecting an account ends its sessions and OAuth grants at once and suspends the organizations it alone belongs to.
  • Accounts: addresses in SUPER_ADMIN_EMAILS become super admins only after they are verified; sign-in returns EMAIL_NOT_VERIFIED until then.
  • Self-hosting: LLM_MODEL_ROTATION=round_robin spreads requests across the configured models.
  • Dashboard: fixed header and sidebar, and loading placeholders in place of spinners.
  • SDKs: the research result types include cached, hops, snippet sources and phase timings.

v1.5.0 (2026-10-10)

  • New POST /orgs/{id}/research: answers (quick) and reports (deep) from the web with verified citations, or structured data from a JSON Schema. Also available as an MCP tool.
  • Search: several engines (Brave and Serper) are queried together and their rankings merged, results are cached, and near-duplicates are removed.
  • Search: search_depth: "advanced" returns each result's relevant passages and a relevance_score; auto_parameters infers sources and freshness from the query; include_answer adds a short cited answer.
  • New Python SDK (xdive, with XDive and AsyncXDive) and JavaScript SDK (xdive), with LangChain tools and a LlamaIndex reader.
  • New GET /whoami: the organization, scopes and plan of an API key, used by the SDKs to find the organization.
  • New public GET /status summary for status pages and monitors.
  • Language-model calls retry transient failures and fall back to LLM_FALLBACK_MODELS; a failing search engine is skipped.

v1.4.0 (2026-10-09)

  • Crawl status responses carry an ETag; sending it in If-None-Match returns 304 while nothing has changed. Running crawls suggest a polling interval with Retry-After: 2.
  • Synchronous endpoints enforce a per-organization concurrency limit (429 CONCURRENCY_LIMIT) and per-server capacity (503 SERVER_BUSY), both with Retry-After.
  • Crawl workers are shared fairly between organizations by plan, and browser-rendered crawls run in their own queue.
  • Synchronous scrape is now checked against the page quota and counts one page per successful scrape.
  • max_age can accept older cached content as well as require fresher content; simultaneous requests for the same uncached page share one fetch.
  • Security: the page cache is keyed by request headers as well as URL, so a page fetched with cookies or authorization headers is no longer served to requests sent without them.
  • With billing configured, PUT /orgs/{id}/plan only downgrades to free; paid plans are activated through checkout.

v1.3.1 (2026-10-09)

  • Security: headless Chrome traffic passes through the SSRF guard.
  • Unreachable hosts fail fast with a clear error instead of escalating to the browser, and the scrape 502 message includes the page's error.

v1.3.0 (2026-10-08)

  • MCP clients such as ChatGPT, claude.ai, Claude Code and Codex connect with OAuth 2.1. Connected clients are listed, and can be revoked, under API Keys → Connected apps.
  • Breaking: the MCP route with an API key in the URL (/api/mcp/{key}) is removed. Clients that send headers keep using API keys.

v1.2.1 (2026-10-08)

  • Fixed: POST /api/mcp no longer answers with a 301 redirect, which broke header-authenticated MCP clients.

v1.2.0 (2026-10-07)

  • MCP connectors for ChatGPT and claude.ai, and a new fetch tool that returns a page's full Markdown.
  • GET and DELETE on the MCP endpoint return 405.
  • Redesigned landing and sign-in pages with light and dark themes.

v1.1.1 (2026-10-07)

  • Every default fetch path escalates past WAF bot filters with Chrome TLS impersonation, so protected sites return content instead of 403 or 404. adaptive_fetch: false still sends a single plain request.

v1.1.0 (2026-10-07)

  • A public landing page for signed-out visitors.

v1.0.3 (2026-10-07)

  • Per-organization requests-per-minute budgets by plan (60, 600 and 6,000) on work-creating endpoints, reported in X-RateLimit-Limit and X-RateLimit-Remaining. Self-hosted servers can turn them off with ORG_RATE_LIMITS=off.

v1.0.0 (2026-10-07)

First release.

  • Scrape in many formats (Markdown, HTML, links, text, screenshots, JSON and more), crawl with webhooks, batch scrape, map, extract, llms.txt, document parsing, and web, news and image search.
  • Adaptive fetching (plain HTTP, TLS impersonation, headless browser), interactive browser sessions, PII redaction, and zero-data-retention and lockdown modes.
  • Organizations, API keys with scopes, expiry and IP allowlists, usage metering with plan quotas, Stripe billing and an audit log.
  • Schedules with change monitoring, an MCP server (stdio and Streamable HTTP), the dashboard, and self-hosting with Docker Compose.