# Rust SDK

The official Rust client for the [XDive](https://xdive.dev) web
data API: scrape, crawl, map, search, research and extract. Async, built on
`reqwest` with `rustls` (no OpenSSL).

## Install

```sh
cargo add xdive
cargo add tokio --features macros,rt-multi-thread
```

## Configuration

| Setting  | Builder method | Environment variable                     |
| -------- | -------------- | ---------------------------------------- |
| API key  | `api_key`      | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`)   |
| Base URL | `base_url`     | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`) |

Builder values win over the environment. The default base URL is
`xdive::DEFAULT_BASE_URL`. One API key identifies one organization; the client
looks the organization id up once with `GET /whoami` and caches it (skip the
lookup with `.org_id(..)`). Other builder methods: `http_client`, `max_retries`
(default 3) and `timeout` (per attempt, default 120 seconds).

```rust,no_run
use std::time::Duration;
use xdive::Client;

// From the environment (XDIVE_API_KEY, optionally XDIVE_BASE_URL)...
let client = Client::from_env()?;

// ...or explicitly.
let client = Client::builder()
    .api_key("ck_live_...")
    .max_retries(5)
    .timeout(Duration::from_secs(60))
    .build()?;
```

## Quickstart

```rust,no_run
use xdive::{Client, CrawlOptions, ResearchOptions, ScrapeOptions, SearchOptions};

#[tokio::main]
async fn main() -> Result<(), xdive::Error> {
    let client = Client::from_env()?;

    // Scrape one page.
    let page = client
        .scrape("https://example.com", ScrapeOptions::new().formats(["markdown"]))
        .await?;
    println!("{}", xdive::page_markdown(&page));

    // Crawl a site and wait for it to finish.
    let record = client
        .crawl(
            "https://example.com",
            CrawlOptions::new().formats(["markdown"]).field("limit", 20),
        )
        .await?;
    println!("{} {}", record.status, record.results.len());

    // Search the web.
    let found = client
        .search("rust async runtimes", SearchOptions::new().limit(5))
        .await?;
    for hit in &found.results {
        println!("{} {}", hit.title, hit.url);
    }

    // Research a question with verified citations.
    let answer = client
        .research("What changed in Rust 1.80?", ResearchOptions::new())
        .await?;
    println!("{} ({})", answer.answer.unwrap_or_default(), answer.confidence);
    for c in &answer.citations {
        println!("[{}] {}", c.number, c.url);
    }
    Ok(())
}
```

Long-running jobs can be started and followed separately:

```rust,no_run
use std::time::Duration;
use xdive::{Client, CrawlOptions, Error, WaitOptions};

let id = client.start_crawl("https://example.com", CrawlOptions::new()).await?;
match client
    .wait_for_crawl(&id, WaitOptions::new().timeout(Duration::from_secs(60)))
    .await
{
    Ok(record) => println!("{} pages", record.results.len()),
    // The crawl keeps running on the server; check on it later.
    Err(Error::CrawlTimeout { crawl_id }) => println!("still running: {crawl_id}"),
    Err(e) => return Err(e),
}
```

Other methods: `start_batch_scrape` / `batch_scrape`, `get_crawl`,
`cancel_crawl`, `crawl_errors`, `map`, `extract`, `llms_txt` and `whoami`.
Request options take an `extra(..)` / `field(key, value)` escape hatch for API
fields the SDK does not model, and results keep unknown fields in `extra`.

## Output formats

Plain formats are names: `.formats(["markdown", "links"])`. Formats with
options are `Format` values:

```rust
use xdive::{Format, ScrapeOptions};

let opts = ScrapeOptions::new().formats([
    Format::markdown(),
    Format::json().prompt("Extract the product name and price"),
]);
```

## Error handling

Every method returns `Result<T, xdive::Error>`.

```rust,no_run
use xdive::{Client, Error, ScrapeOptions};

match client.scrape("https://example.com", ScrapeOptions::new()).await {
    Ok(page) => println!("{}", page.title.unwrap_or_default()),
    Err(Error::RateLimited { error, retry_after }) => {
        eprintln!("slow down ({}), retry in {retry_after:?}", error.message);
    }
    Err(Error::QuotaExceeded(e)) => eprintln!("plan quota used up: {}", e.message),
    Err(Error::Api(e)) => eprintln!(
        "API error {} {:?}: {} (request {:?})",
        e.status, e.code, e.message, e.request_id
    ),
    Err(Error::Transport(e)) => eprintln!("network problem: {e}"),
    Err(e) => eprintln!("{e}"),
}
```

`Error::is_retryable()` tells whether repeating the request later may work and
`Error::status()` returns the HTTP status when the error came from a response.
`ApiError` carries `status`, `code`, `message`, `retryable`, `request_id` and
`docs_url`.

## Retries and idempotency

The client retries, up to `max_retries` times (default 3):

- HTTP 429, 503 with code `SERVER_BUSY` and 409 with code
  `IDEMPOTENCY_KEY_IN_PROGRESS`, after waiting the `Retry-After` header
  (default 1 second, fractional seconds allowed). When the budget runs out the
  result is `Error::RateLimited`.
- Transport errors and timeouts, for GET and POST, with a backoff of 1s, 2s,
  4s, ... capped at 8s.

Other statuses are never retried. Every POST carries an `Idempotency-Key` header
(a random UUID v4, generated once per call and reused by each retry), so a
retry after a timeout cannot run the work twice. Pass your own key to make a
call safe to repeat across processes:

```rust,no_run
use xdive::{Client, CrawlOptions};

let id = client
    .start_crawl(
        "https://example.com",
        CrawlOptions::new().idempotency_key("nightly-2026-10-18"),
    )
    .await?;
```

Dropping a future (for example with `tokio::time::timeout` or `select!`) cancels
the call cleanly, including any retry wait.

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-rust · All docs for agents: https://xdive.dev/llms.txt