# .NET SDK

The official .NET client for the XDive web data API: scrape, crawl, map, search, research and
extract web pages as clean Markdown and structured data.

- Targets `netstandard2.0` and `net8.0`: .NET Framework 4.6.2+, .NET 6/7/8/9, Unity, Xamarin.
- Async everywhere, `CancellationToken` last, nullable reference types, XML docs.
- Safe retries: `Retry-After` is honored and every POST carries an `Idempotency-Key`.
- Works with `IHttpClientFactory` and dependency injection.

## Install

```sh
dotnet add package XDive
```

## Quickstart

```csharp
using XDive;

using var client = XDiveClient.FromEnvironment(); // reads XDIVE_API_KEY

// Scrape one page.
Page page = await client.ScrapeAsync("https://example.com",
    new ScrapeOptions { Formats = Format.Of("markdown") });
Console.WriteLine(XDiveClient.PageMarkdown(page));

// Crawl a site and wait for it to finish.
CrawlRecord crawl = await client.CrawlAsync("https://example.com", new CrawlOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["limit"] = 20 },
});
Console.WriteLine($"{crawl.Status}: {crawl.Results.Count} pages");

// Search the web.
SearchResponse found = await client.SearchAsync("dotnet cancellation tokens",
    new SearchOptions { Limit = 5 });
foreach (var r in found.Results) Console.WriteLine($"{r.Title} {r.Url}");

// Research a question with verified citations.
ResearchResult answer = await client.ResearchAsync("What changed in .NET 8?");
Console.WriteLine(answer.Answer);
foreach (var c in answer.Citations) Console.WriteLine($"[{c.Number}] {c.Url}");
```

Other calls: `StartCrawlAsync` / `GetCrawlAsync` / `WaitForCrawlAsync` / `CancelCrawlAsync` /
`CrawlErrorsAsync`, `StartBatchScrapeAsync` / `BatchScrapeAsync` (up to 100 URLs), `MapAsync`,
`ExtractAsync`, `LlmsTxtAsync`, `WhoamiAsync`.

### Formats

`Format.Of("markdown", "links")` makes plain formats. A string converts implicitly, and object
formats carry their options:

```csharp
Formats = new List<Format>
{
    "markdown",
    Format.Json(new Dictionary<string, object?> { ["type"] = "object" }, prompt: "the product price"),
    new Format("highlights") { Query = "pricing", MaxResults = 5 },
}
```

### Fields the SDK does not model

Every options class has an `Extra` dictionary that is merged into the request body (and wins over
the typed fields). Every result keeps the complete response as `Raw` (a `JsonElement`), and unknown
JSON fields never break parsing:

```csharp
var page = await client.ScrapeAsync(url, new ScrapeOptions
{
    Formats = Format.Of("markdown"),
    Extra = new Dictionary<string, object?> { ["only_main_content"] = true },
});
var surprise = page.Raw.GetProperty("someNewField");
```

## Configuration

| Setting  | Option                | Environment variable                     |
| -------- | --------------------- | ---------------------------------------- |
| API key  | `XDiveOptions.ApiKey`  | `XDIVE_API_KEY` (or `CRAWLER_API_KEY`)   |
| Base URL | `XDiveOptions.BaseUrl` | `XDIVE_BASE_URL` (or `CRAWLER_BASE_URL`) |

Options win over the environment. The default base URL is `XDiveClient.DefaultBaseUrl`.

```csharp
var client = new XDiveClient(new XDiveOptions
{
    ApiKey = "ck_live_...",
    BaseUrl = "https://your-host/api",   // optional
    OrgId = "org_123",                    // optional: skips the GET /whoami lookup
    MaxRetries = 3,                       // default 3, 0 disables retries
    Timeout = TimeSpan.FromSeconds(120),  // per attempt, default 120s
    HttpClient = myHttpClient,            // optional, see below
});
```

One API key identifies one organization. The client looks the organization id up once with
`GET /whoami` and caches it (thread-safe); set `OrgId` to skip the lookup.

## Dependency injection

`XDiveClient` is thread-safe: register one instance. The optional package
`XDive.Extensions.DependencyInjection` does it on top of `IHttpClientFactory`:

```sh
dotnet add package XDive.Extensions.DependencyInjection
```

```csharp
builder.Services.AddXDive(o => o.ApiKey = builder.Configuration["XDive:ApiKey"]);

public class MyService(XDiveClient xdive)
{
    public Task<Page> Fetch(string url, CancellationToken ct) => xdive.ScrapeAsync(url, null, ct);
}
```

`AddXDive` returns the `IHttpClientBuilder`, so you can add handlers or resilience policies.

Without the extension package, register it yourself:

```csharp
builder.Services.AddHttpClient("xdive");
builder.Services.AddSingleton(sp => new XDiveClient(new XDiveOptions
{
    ApiKey = builder.Configuration["XDive:ApiKey"],
    HttpClient = sp.GetRequiredService<IHttpClientFactory>().CreateClient("xdive"),
}));
```

When you pass your own `HttpClient`, the client never disposes it and the `HttpClient.Timeout`
still applies (default 100s): raise it above `XDiveOptions.Timeout` for long calls such as
`ResearchAsync(..., new ResearchOptions { Depth = "deep" })`. `XDiveClient.Dispose()` only disposes
an `HttpClient` the client created itself.

## Errors

API errors are thrown as `XDiveException`:

```csharp
try
{
    await client.ScrapeAsync("https://example.com");
}
catch (QuotaExceededException e)          // HTTP 402: plan quota exhausted
{
    Console.WriteLine($"Upgrade your plan ({e.RequestId})");
}
catch (RateLimitedException e)            // retries ran out on 429 / SERVER_BUSY
{
    await Task.Delay(e.RetryAfter);
}
catch (XDiveException e)
{
    Console.WriteLine($"{e.Status} {e.Code}: {e.Message}");
    Console.WriteLine($"retryable={e.Retryable} request={e.RequestId} docs={e.DocsUrl}");
}
```

| Property    | Meaning                                                          |
| ----------- | ---------------------------------------------------------------- |
| `Status`    | HTTP status                                                      |
| `Code`      | machine-readable code, `null` for older error bodies             |
| `Retryable` | the server's `retryable` flag                                    |
| `RequestId` | the `X-Request-ID` header, quote it to support                   |
| `DocsUrl`   | documentation link for the error, when provided                  |

`CrawlTimeoutException` (with `CrawlId`) is thrown by `WaitForCrawlAsync` / `CrawlAsync` when the
crawl does not finish within `Timeout` (default 10 minutes); the crawl keeps running on the server.
A request that times out on every attempt throws `TimeoutException`; cancelling the token throws
`OperationCanceledException`.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS` are retried
  after `Retry-After` seconds (default 1, fractions allowed), up to `MaxRetries` times, then
  `RateLimitedException` is thrown.
- Transport errors and timeouts are retried for GET and POST with backoff 1s, 2s, 4s, capped at 8s.
- Other statuses are never retried.
- Every POST carries an `Idempotency-Key` header (a random GUID) generated once per call and reused
  by every retry, so a retry after a timeout never runs the work twice. Set your own with
  `new ScrapeOptions { IdempotencyKey = "order-1234" }`; calls with the same key return the first
  call's response.
- Every wait honors the `CancellationToken`.

## Waiting for crawls

`WaitForCrawlAsync` sends `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and it
follows the server's `Retry-After`:

```csharp
string id = await client.StartCrawlAsync("https://example.com");
CrawlRecord done = await client.WaitForCrawlAsync(id,
    new WaitOptions { Timeout = TimeSpan.FromMinutes(5) }, cancellationToken);
CrawlErrors errors = await client.CrawlErrorsAsync(id);
```

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-dotnet · All docs for agents: https://xdive.dev/llms.txt