.NET SDK

The official .NET client for the XDive web data API: scrape, crawl, map, search, research and extract web pages as clean Markdown and structured data.

Targets netstandard2.0 and net8.0: .NET Framework 4.6.2+, .NET 6/7/8/9, Unity, Xamarin.
Async everywhere, CancellationToken last, nullable reference types, XML docs.
Safe retries: Retry-After is honored and every POST carries an Idempotency-Key.
Works with IHttpClientFactory and dependency injection.

Install

dotnet add package XDive

Quickstart

csharp

using XDive;
 
using var client = XDiveClient.FromEnvironment(); // reads XDIVE_API_KEY
 
// Scrape one page.
Page page = await client.ScrapeAsync("https://example.com",
    new ScrapeOptions { Formats = Format.Of("markdown") });
Console.WriteLine(XDiveClient.PageMarkdown(page));
 
// Crawl a site and wait for it to finish.
CrawlRecord crawl = await client.CrawlAsync("https://example.com", new CrawlOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["limit"] = 20 },
});
Console.WriteLine(
quot;{crawl.Status}: {crawl.Results.Count} pages");
 
// Search the web.
SearchResponse found = await client.SearchAsync("dotnet cancellation tokens",
    new SearchOptions { Limit = 5 });
foreach (var r in found.Results) Console.WriteLine(
quot;{r.Title} {r.Url}");
 
// Research a question with verified citations.
ResearchResult answer = await client.ResearchAsync("What changed in .NET 8?");
Console.WriteLine(answer.Answer);
foreach (var c in answer.Citations) Console.WriteLine(
quot;[{c.Number}] {c.Url}");

Other calls: StartCrawlAsync / GetCrawlAsync / WaitForCrawlAsync / CancelCrawlAsync / CrawlErrorsAsync, StartBatchScrapeAsync / BatchScrapeAsync (up to 100 URLs), MapAsync, ExtractAsync, LlmsTxtAsync, WhoamiAsync.

Formats

Format.Of("markdown", "links") makes plain formats. A string converts implicitly, and object formats carry their options:

csharp

Formats = new List<Format>
{
    "markdown",
    Format.Json(new Dictionary<string, object?> { ["type"] = "object" }, prompt: "the product price"),
    new Format("highlights") { Query = "pricing", MaxResults = 5 },
}

Fields the SDK does not model

Every options class has an Extra dictionary that is merged into the request body (and wins over the typed fields). Every result keeps the complete response as Raw (a JsonElement), and unknown JSON fields never break parsing:

csharp

var page = await client.ScrapeAsync(url, new ScrapeOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["only_main_content"] = true },
});
var surprise = page.Raw.GetProperty("someNewField");

Configuration

Setting	Option	Environment variable
API key	`XDiveOptions.ApiKey`	`XDIVE_API_KEY` (or `CRAWLER_API_KEY`)
Base URL	`XDiveOptions.BaseUrl`	`XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`)

Options win over the environment. The default base URL is XDiveClient.DefaultBaseUrl.

csharp

var client = new XDiveClient(new XDiveOptions
{
    ApiKey = "ck_live_...",
    BaseUrl = "https://your-host/api",   // optional
    OrgId = "org_123",                    // optional: skips the GET /whoami lookup
    MaxRetries = 3,                       // default 3, 0 disables retries
    Timeout = TimeSpan.FromSeconds(120),  // per attempt, default 120s
    HttpClient = myHttpClient,            // optional, see below
});

One API key identifies one organization. The client looks the organization id up once with GET /whoami and caches it (thread-safe); set OrgId to skip the lookup.

Dependency injection

XDiveClient is thread-safe: register one instance. The optional package XDive.Extensions.DependencyInjection does it on top of IHttpClientFactory:

dotnet add package XDive.Extensions.DependencyInjection

csharp

builder.Services.AddXDive(o => o.ApiKey = builder.Configuration["XDive:ApiKey"]);
 
public class MyService(XDiveClient xdive)
{
    public Task<Page> Fetch(string url, CancellationToken ct) => xdive.ScrapeAsync(url, null, ct);
}

AddXDive returns the IHttpClientBuilder, so you can add handlers or resilience policies.

Without the extension package, register it yourself:

csharp

builder.Services.AddHttpClient("xdive");
builder.Services.AddSingleton(sp => new XDiveClient(new XDiveOptions
{
    ApiKey = builder.Configuration["XDive:ApiKey"],
    HttpClient = sp.GetRequiredService<IHttpClientFactory>().CreateClient("xdive"),
}));

When you pass your own HttpClient, the client never disposes it and the HttpClient.Timeout still applies (default 100s): raise it above XDiveOptions.Timeout for long calls such as ResearchAsync(..., new ResearchOptions { Depth = "deep" }). XDiveClient.Dispose() only disposes an HttpClient the client created itself.

Errors

API errors are thrown as XDiveException:

csharp

try
{
    await client.ScrapeAsync("https://example.com");
}
catch (QuotaExceededException e)          // HTTP 402: plan quota exhausted
{
    Console.WriteLine(
quot;Upgrade your plan ({e.RequestId})");
}
catch (RateLimitedException e)            // retries ran out on 429 / SERVER_BUSY
{
    await Task.Delay(e.RetryAfter);
}
catch (XDiveException e)
{
    Console.WriteLine(
quot;{e.Status} {e.Code}: {e.Message}");
    Console.WriteLine(
quot;retryable={e.Retryable} request={e.RequestId} docs={e.DocsUrl}");
}

Property	Meaning
`Status`	HTTP status
`Code`	machine-readable code, `null` for older error bodies
`Retryable`	the server's `retryable` flag
`RequestId`	the `X-Request-ID` header, quote it to support
`DocsUrl`	documentation link for the error, when provided

CrawlTimeoutException (with CrawlId) is thrown by WaitForCrawlAsync / CrawlAsync when the crawl does not finish within Timeout (default 10 minutes); the crawl keeps running on the server. A request that times out on every attempt throws TimeoutException; cancelling the token throws OperationCanceledException.

Retries and idempotency

429, 503 with code SERVER_BUSY and 409 with code IDEMPOTENCY_KEY_IN_PROGRESS are retried after Retry-After seconds (default 1, fractions allowed), up to MaxRetries times, then RateLimitedException is thrown.
Transport errors and timeouts are retried for GET and POST with backoff 1s, 2s, 4s, capped at 8s.
Other statuses are never retried.
Every POST carries an Idempotency-Key header (a random GUID) generated once per call and reused by every retry, so a retry after a timeout never runs the work twice. Set your own with new ScrapeOptions { IdempotencyKey = "order-1234" }; calls with the same key return the first call's response.
Every wait honors the CancellationToken.

Waiting for crawls

WaitForCrawlAsync sends If-None-Match, so an unchanged crawl costs a bodiless 304, and it follows the server's Retry-After:

csharp

string id = await client.StartCrawlAsync("https://example.com");
CrawlRecord done = await client.WaitForCrawlAsync(id,
    new WaitOptions { Timeout = TimeSpan.FromMinutes(5) }, cancellationToken);
CrawlErrors errors = await client.CrawlErrorsAsync(id);

.NET SDK

Install

Quickstart

Formats

Fields the SDK does not model

Configuration

Dependency injection

Errors

Retries and idempotency

Waiting for crawls

See also