Rust SDK

The official Rust client for the XDive web data API: scrape, crawl, map, search, research and extract. Async, built on reqwest with rustls (no OpenSSL).

Install

cargo add xdive
cargo add tokio --features macros,rt-multi-thread

Configuration

Setting	Builder method	Environment variable
API key	`api_key`	`XDIVE_API_KEY` (or `CRAWLER_API_KEY`)
Base URL	`base_url`	`XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`)

Builder values win over the environment. The default base URL is xdive::DEFAULT_BASE_URL. One API key identifies one organization; the client looks the organization id up once with GET /whoami and caches it (skip the lookup with .org_id(..)). Other builder methods: http_client, max_retries (default 3) and timeout (per attempt, default 120 seconds).

rust,no_run

use std::time::Duration;
use xdive::Client;
 
// From the environment (XDIVE_API_KEY, optionally XDIVE_BASE_URL)...
let client = Client::from_env()?;
 
// ...or explicitly.
let client = Client::builder()
    .api_key("ck_live_...")
    .max_retries(5)
    .timeout(Duration::from_secs(60))
    .build()?;

Quickstart

rust,no_run

use xdive::{Client, CrawlOptions, ResearchOptions, ScrapeOptions, SearchOptions};
 
#[tokio::main]
async fn main() -> Result<(), xdive::Error> {
    let client = Client::from_env()?;
 
    // Scrape one page.
    let page = client
        .scrape("https://example.com", ScrapeOptions::new().formats(["markdown"]))
        .await?;
    println!("{}", xdive::page_markdown(&page));
 
    // Crawl a site and wait for it to finish.
    let record = client
        .crawl(
            "https://example.com",
            CrawlOptions::new().formats(["markdown"]).field("limit", 20),
        )
        .await?;
    println!("{} {}", record.status, record.results.len());
 
    // Search the web.
    let found = client
        .search("rust async runtimes", SearchOptions::new().limit(5))
        .await?;
    for hit in &found.results {
        println!("{} {}", hit.title, hit.url);
    }
 
    // Research a question with verified citations.
    let answer = client
        .research("What changed in Rust 1.80?", ResearchOptions::new())
        .await?;
    println!("{} ({})", answer.answer.unwrap_or_default(), answer.confidence);
    for c in &answer.citations {
        println!("[{}] {}", c.number, c.url);
    }
    Ok(())
}

Long-running jobs can be started and followed separately:

rust,no_run

use std::time::Duration;
use xdive::{Client, CrawlOptions, Error, WaitOptions};
 
let id = client.start_crawl("https://example.com", CrawlOptions::new()).await?;
match client
    .wait_for_crawl(&id, WaitOptions::new().timeout(Duration::from_secs(60)))
    .await
{
    Ok(record) => println!("{} pages", record.results.len()),
    // The crawl keeps running on the server; check on it later.
    Err(Error::CrawlTimeout { crawl_id }) => println!("still running: {crawl_id}"),
    Err(e) => return Err(e),
}

Other methods: start_batch_scrape / batch_scrape, get_crawl, cancel_crawl, crawl_errors, map, extract, llms_txt and whoami. Request options take an extra(..) / field(key, value) escape hatch for API fields the SDK does not model, and results keep unknown fields in extra.

Output formats

Plain formats are names: .formats(["markdown", "links"]). Formats with options are Format values:

rust

use xdive::{Format, ScrapeOptions};
 
let opts = ScrapeOptions::new().formats([
    Format::markdown(),
    Format::json().prompt("Extract the product name and price"),
]);

Error handling

Every method returns Result<T, xdive::Error>.

rust,no_run

use xdive::{Client, Error, ScrapeOptions};
 
match client.scrape("https://example.com", ScrapeOptions::new()).await {
    Ok(page) => println!("{}", page.title.unwrap_or_default()),
    Err(Error::RateLimited { error, retry_after }) => {
        eprintln!("slow down ({}), retry in {retry_after:?}", error.message);
    }
    Err(Error::QuotaExceeded(e)) => eprintln!("plan quota used up: {}", e.message),
    Err(Error::Api(e)) => eprintln!(
        "API error {} {:?}: {} (request {:?})",
        e.status, e.code, e.message, e.request_id
    ),
    Err(Error::Transport(e)) => eprintln!("network problem: {e}"),
    Err(e) => eprintln!("{e}"),
}

Error::is_retryable() tells whether repeating the request later may work and Error::status() returns the HTTP status when the error came from a response. ApiError carries status, code, message, retryable, request_id and docs_url.

Retries and idempotency

The client retries, up to max_retries times (default 3):

HTTP 429, 503 with code SERVER_BUSY and 409 with code IDEMPOTENCY_KEY_IN_PROGRESS, after waiting the Retry-After header (default 1 second, fractional seconds allowed). When the budget runs out the result is Error::RateLimited.
Transport errors and timeouts, for GET and POST, with a backoff of 1s, 2s, 4s, ... capped at 8s.

Other statuses are never retried. Every POST carries an Idempotency-Key header (a random UUID v4, generated once per call and reused by each retry), so a retry after a timeout cannot run the work twice. Pass your own key to make a call safe to repeat across processes:

rust,no_run

use xdive::{Client, CrawlOptions};
 
let id = client
    .start_crawl(
        "https://example.com",
        CrawlOptions::new().idempotency_key("nightly-2026-10-18"),
    )
    .await?;

Dropping a future (for example with tokio::time::timeout or select!) cancels the call cleanly, including any retry wait.

Rust SDK

Install

Configuration

Quickstart

Output formats

Error handling

Retries and idempotency

See also