Java SDK

Official Java client for the XDive web-data API: scrape, crawl, map, search, research and extract. Requires Java 11 or newer, works from Kotlin and Scala, and depends only on Jackson databind (HTTP is java.net.http.HttpClient).

Install

Maven:

xml

<dependency>
  <groupId>dev.xdive</groupId>
  <artifactId>xdive</artifactId>
  <version>1.5.0</version>
</dependency>

Gradle (Kotlin DSL):

kotlin

dependencies {
    implementation("dev.xdive:xdive:1.5.0")
}

Quickstart

java

import dev.xdive.*;
import java.util.List;
 
public class Quickstart {
    public static void main(String[] args) {
        // Reads XDIVE_API_KEY (and XDIVE_BASE_URL) from the environment.
        XDive xdive = XDive.builder().build();
 
        // Scrape one page.
        Page page = xdive.scrape("https://example.com",
                ScrapeOptions.builder().formats("markdown", "links").build());
        System.out.println(XDive.pageMarkdown(page));
 
        // Crawl a site and wait for the result.
        CrawlRecord crawl = xdive.crawl("https://example.com",
                CrawlOptions.builder().formats("markdown").extra("limit", 20).build());
        for (Page p : crawl.getResults()) {
            System.out.println(p.getUrl() + " " + XDive.pageMarkdown(p).length());
        }
 
        // Search the web.
        SearchResponse results = xdive.search("open source web crawlers",
                SearchOptions.builder().limit(5).build());
        results.getResults().forEach(r -> System.out.println(r.getTitle() + " " + r.getUrl()));
 
        // Research a question with citations.
        ResearchResult answer = xdive.research("What is robots.txt?");
        System.out.println(answer.getAnswer());
        answer.getCitations().forEach(c -> System.out.println("[" + c.getNumber() + "] " + c.getUrl()));
    }
}

Other calls: startCrawl, getCrawl, waitForCrawl, cancelCrawl, crawlErrors, startBatchScrape, batchScrape, map, extract, llmsTxt, whoami.

Every main call has a CompletableFuture variant (scrapeAsync, searchAsync, researchAsync, crawlAsync, ...). A failed future completes with a CompletionException whose cause is the XDiveException.

Formats

Plain formats are strings (formats("markdown", "links")). Formats with options are objects:

java

Format json = Format.builder("json")
        .prompt("Extract the product name and price")
        .schema(Map.of("type", "object"))
        .build();
Page p = xdive.scrape(url, ScrapeOptions.builder().formats(Format.of("markdown"), json).build());

XDive.pageMarkdown(page) returns the Markdown whichever field carries it.

Fields the SDK does not model yet

Every options builder has extra(Map) / extra(key, value), merged into the request body (and winning over modeled fields). Every result keeps the complete response in getRaw() (a Jackson JsonNode), and unknown fields never break parsing.

Configuration

Setting	Builder	Environment	Default
API key	`apiKey(..)`	`XDIVE_API_KEY`, then `CRAWLER_API_KEY`	required
API origin	`baseUrl(..)`	`XDIVE_BASE_URL`, then `CRAWLER_BASE_URL`	`XDive.DEFAULT_BASE_URL`
Organization id	`orgId(..)`	none	discovered once via `GET /whoami`, cached
Retries	`maxRetries(n)`	none	3
Per-attempt timeout	`timeout(Duration)`	none	120 s
HTTP client	`httpClient(..)`	none	`HttpClient.newHttpClient()`

The client is thread-safe: create one and share it.

Errors

Failures are unchecked XDiveExceptions:

java

try {
    xdive.scrape("https://example.com");
} catch (RateLimitedException e) {
    // The API kept saying "slow down" until the retries ran out.
    System.out.println("try again in " + e.getRetryAfter());
} catch (QuotaExceededException e) {
    // HTTP 402: the plan quota is used up.
} catch (XDiveException e) {
    System.out.println(e.getStatus() + " " + e.getCode() + ": " + e.getMessage());
    System.out.println("retryable=" + e.isRetryable()
            + " requestId=" + e.getRequestId() + " docs=" + e.getDocsUrl());
}

CrawlTimeoutException (HTTP status 408, getCrawlId()) is thrown when waitForCrawl / crawl / batchScrape times out; the crawl keeps running on the server. Transport failures that outlast the retries surface as an XDiveException with status 0 and code NETWORK_ERROR. A missing API key makes build() throw IllegalStateException.

Retries and idempotency

429, 503 with code SERVER_BUSY and 409 with code IDEMPOTENCY_KEY_IN_PROGRESS are retried after the Retry-After seconds (default 1, fractions allowed), up to maxRetries, then RateLimitedException.
Transport errors and timeouts are retried for GET and POST with a 1 s, 2 s, 4 s ... backoff capped at 8 s.
No other status is retried.
Every POST carries an Idempotency-Key header (a random UUID generated once per call and reused by each retry), so a retry never runs or bills the work twice. A new call gets a new key. To set your own: ScrapeOptions.builder().idempotencyKey("order-1234-scrape").build().

Kotlin

kotlin

import dev.xdive.CrawlOptions
import dev.xdive.XDive
 
fun main() {
    val xdive = XDive.builder().apiKey(System.getenv("XDIVE_API_KEY")).build()
 
    val page = xdive.scrape("https://example.com")
    println(XDive.pageMarkdown(page))
 
    val crawl = xdive.crawl(
        "https://example.com",
        CrawlOptions.builder().formats("markdown").extra("limit", 10).build(),
    )
    crawl.results.forEach { println(it.url) }
}

Java SDK

Install

Quickstart

Formats

Fields the SDK does not model yet

Configuration

Errors

Retries and idempotency

Kotlin

See also