# Java SDK

Official Java client for the [XDive](https://xdive.dev) web-data
API: scrape, crawl, map, search, research and extract. Requires Java 11 or newer,
works from Kotlin and Scala, and depends only on Jackson databind
(HTTP is `java.net.http.HttpClient`).

## Install

Maven:

```xml
<dependency>
  <groupId>dev.xdive</groupId>
  <artifactId>xdive</artifactId>
  <version>1.5.0</version>
</dependency>
```

Gradle (Kotlin DSL):

```kotlin
dependencies {
    implementation("dev.xdive:xdive:1.5.0")
}
```

## Quickstart

```java
import dev.xdive.*;
import java.util.List;

public class Quickstart {
    public static void main(String[] args) {
        // Reads XDIVE_API_KEY (and XDIVE_BASE_URL) from the environment.
        XDive xdive = XDive.builder().build();

        // Scrape one page.
        Page page = xdive.scrape("https://example.com",
                ScrapeOptions.builder().formats("markdown", "links").build());
        System.out.println(XDive.pageMarkdown(page));

        // Crawl a site and wait for the result.
        CrawlRecord crawl = xdive.crawl("https://example.com",
                CrawlOptions.builder().formats("markdown").extra("limit", 20).build());
        for (Page p : crawl.getResults()) {
            System.out.println(p.getUrl() + " " + XDive.pageMarkdown(p).length());
        }

        // Search the web.
        SearchResponse results = xdive.search("open source web crawlers",
                SearchOptions.builder().limit(5).build());
        results.getResults().forEach(r -> System.out.println(r.getTitle() + " " + r.getUrl()));

        // Research a question with citations.
        ResearchResult answer = xdive.research("What is robots.txt?");
        System.out.println(answer.getAnswer());
        answer.getCitations().forEach(c -> System.out.println("[" + c.getNumber() + "] " + c.getUrl()));
    }
}
```

Other calls: `startCrawl`, `getCrawl`, `waitForCrawl`, `cancelCrawl`, `crawlErrors`,
`startBatchScrape`, `batchScrape`, `map`, `extract`, `llmsTxt`, `whoami`.

Every main call has a `CompletableFuture` variant (`scrapeAsync`, `searchAsync`,
`researchAsync`, `crawlAsync`, ...). A failed future completes with a
`CompletionException` whose cause is the `XDiveException`.

### Formats

Plain formats are strings (`formats("markdown", "links")`). Formats with options
are objects:

```java
Format json = Format.builder("json")
        .prompt("Extract the product name and price")
        .schema(Map.of("type", "object"))
        .build();
Page p = xdive.scrape(url, ScrapeOptions.builder().formats(Format.of("markdown"), json).build());
```

`XDive.pageMarkdown(page)` returns the Markdown whichever field carries it.

### Fields the SDK does not model yet

Every options builder has `extra(Map)` / `extra(key, value)`, merged into the
request body (and winning over modeled fields). Every result keeps the complete
response in `getRaw()` (a Jackson `JsonNode`), and unknown fields never break
parsing.

## Configuration

| Setting | Builder | Environment | Default |
| --- | --- | --- | --- |
| API key | `apiKey(..)` | `XDIVE_API_KEY`, then `CRAWLER_API_KEY` | required |
| API origin | `baseUrl(..)` | `XDIVE_BASE_URL`, then `CRAWLER_BASE_URL` | `XDive.DEFAULT_BASE_URL` |
| Organization id | `orgId(..)` | none | discovered once via `GET /whoami`, cached |
| Retries | `maxRetries(n)` | none | 3 |
| Per-attempt timeout | `timeout(Duration)` | none | 120 s |
| HTTP client | `httpClient(..)` | none | `HttpClient.newHttpClient()` |

The client is thread-safe: create one and share it.

## Errors

Failures are unchecked `XDiveException`s:

```java
try {
    xdive.scrape("https://example.com");
} catch (RateLimitedException e) {
    // The API kept saying "slow down" until the retries ran out.
    System.out.println("try again in " + e.getRetryAfter());
} catch (QuotaExceededException e) {
    // HTTP 402: the plan quota is used up.
} catch (XDiveException e) {
    System.out.println(e.getStatus() + " " + e.getCode() + ": " + e.getMessage());
    System.out.println("retryable=" + e.isRetryable()
            + " requestId=" + e.getRequestId() + " docs=" + e.getDocsUrl());
}
```

`CrawlTimeoutException` (HTTP status 408, `getCrawlId()`) is thrown when
`waitForCrawl` / `crawl` / `batchScrape` times out; the crawl keeps running on the
server. Transport failures that outlast the retries surface as an
`XDiveException` with status `0` and code `NETWORK_ERROR`. A missing API key makes
`build()` throw `IllegalStateException`.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code
  `IDEMPOTENCY_KEY_IN_PROGRESS` are retried after the `Retry-After` seconds
  (default 1, fractions allowed), up to `maxRetries`, then `RateLimitedException`.
- Transport errors and timeouts are retried for GET and POST with a 1 s, 2 s, 4 s
  ... backoff capped at 8 s.
- No other status is retried.
- Every POST carries an `Idempotency-Key` header (a random UUID generated once
  per call and reused by each retry), so a retry never runs or bills the work
  twice. A new call gets a new key. To set your own:
  `ScrapeOptions.builder().idempotencyKey("order-1234-scrape").build()`.

## Kotlin

```kotlin
import dev.xdive.CrawlOptions
import dev.xdive.XDive

fun main() {
    val xdive = XDive.builder().apiKey(System.getenv("XDIVE_API_KEY")).build()

    val page = xdive.scrape("https://example.com")
    println(XDive.pageMarkdown(page))

    val crawl = xdive.crawl(
        "https://example.com",
        CrawlOptions.builder().formats("markdown").extra("limit", 10).build(),
    )
    crawl.results.forEach { println(it.url) }
}
```

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-java · All docs for agents: https://xdive.dev/llms.txt