# PHP SDK

The official PHP SDK for the [XDive](https://xdive.dev) web-data API: scrape, crawl, search and research the web as clean Markdown. PHP 8.1+, works in plain PHP, Laravel and Symfony. HTTP goes through PSR-18, so you can plug in any client; Guzzle is the default.

## Install

```bash
composer require xdive/xdive
```

## Quickstart

```php
<?php

use XDive\Client;
use XDive\Format;

require 'vendor/autoload.php';

$client = new Client(apiKey: getenv('XDIVE_API_KEY'));   // or Client::fromEnvironment()

// Scrape one page
$page = $client->scrape('https://example.com', [Format::MARKDOWN]);
echo $page->title, "\n", $page->markdown(), "\n";

// Crawl a site and wait for the result
$crawl = $client->crawl('https://example.com', [Format::MARKDOWN], ['limit' => 20, 'max_depth' => 2]);
foreach ($crawl->results as $p) {
    echo $p->url, ' ', strlen($p->markdown()), " chars\n";
}

// Search the web
$search = $client->search('php 8.4 release notes', limit: 5, includeAnswer: true);
foreach ($search->results as $hit) {
    echo "{$hit->rank}. {$hit->title} {$hit->url}\n";
}

// Research a question with verified citations
$research = $client->research('How does HTTP/3 differ from HTTP/2?', depth: 'quick');
echo $research->answer, "\n";
foreach ($research->citations as $c) {
    echo "[{$c->number}] {$c->url}\n";
}
```

### Methods

| Method | What it does |
| --- | --- |
| `scrape($url, $formats, $extra)` | One page, synchronously. Returns `Page`. |
| `startCrawl` / `crawl` / `getCrawl` / `waitForCrawl` / `cancelCrawl` / `crawlErrors` | The crawl lifecycle. `crawl()` is start + wait. |
| `startBatchScrape` / `batchScrape` | Up to 100 URLs through the job pipeline (follow with `waitForCrawl`). |
| `map($url, search:, limit:, maxDepth:)` | Discover a site's URLs. Returns `list<string>`. |
| `search($query, ...)` | Web search. Returns `SearchResponse`. |
| `research($query, depth:, output:, schema:)` | Cited answer. Returns `ResearchResult`. |
| `extract($urls, prompt:, schema:)` | Structured data across up to 10 URLs. |
| `llmsTxt($url, maxUrls:, fullText:)` | Generate `llms.txt` for a site. |
| `whoami()` / `orgId()` | The key's organization, scopes and plan. |
| `Client::pageMarkdown($page)` / `$page->markdown()` | Markdown of a page, whichever field carries it. |

All methods are synchronous and return readonly value objects (`Page`, `CrawlRecord`, `SearchResponse`, `ResearchResult`, `Citation`, ...) with `public readonly` properties for the stable fields. Every value object has `raw()`, the decoded array exactly as received, for fields the SDK does not model. Open-ended parts (`extract()`, `llmsTxt()`, `whoami()`) are plain arrays.

### Options

Common options are named arguments; anything else the API accepts goes in the `$extra` array (it wins over named options, `null` values are dropped):

```php
$page = $client->scrape(
    'https://example.com/pricing',
    formats: [
        Format::MARKDOWN,                                     // plain format
        Format::object(Format::JSON, prompt: 'Extract plans', // object format
            schema: ['type' => 'object', 'properties' => ['plans' => ['type' => 'array']]]),
    ],
    extra: ['only_main_content' => true, 'wait_for' => 1000],
);
```

`Format::names('markdown', 'links')` builds a list of plain formats; `Format::object($type, prompt:, schema:, question:, query:, maxResults:, extra:)` builds an object format.

### Waiting for crawls

`waitForCrawl()` polls with `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and follows the server's `Retry-After`. It throws `CrawlTimeoutException` after `$timeout` seconds (default 600); the crawl keeps running on the server.

```php
$id = $client->startCrawl('https://example.com', extra: ['limit' => 100]);
$record = $client->waitForCrawl($id, timeout: 300.0, pollInterval: 2.0);
$failed = $client->crawlErrors($id);   // failed pages and robots.txt-blocked URLs
```

## Configuration

```php
$client = new Client(
    apiKey: 'xd_...',                          // default: XDIVE_API_KEY, then CRAWLER_API_KEY
    baseUrl: Client::DEFAULT_BASE_URL,         // default: XDIVE_BASE_URL, then CRAWLER_BASE_URL
    orgId: 'org_123',                          // skips the one-time GET /whoami discovery
    maxRetries: 3,
    timeout: 120.0,                            // per attempt, default Guzzle client only
    httpClient: $psr18Client,                  // any Psr\Http\Client\ClientInterface
);
```

| Variable | Purpose |
| --- | --- |
| `XDIVE_API_KEY` (fallback `CRAWLER_API_KEY`) | API key |
| `XDIVE_BASE_URL` (fallback `CRAWLER_BASE_URL`) | API origin including `/api` |

The organization id is discovered once with `GET /whoami` and cached on the client instance. The client holds no other state, so create it once and reuse it.

To use another HTTP client, pass any PSR-18 client; request and stream factories default to `guzzlehttp/psr7` and can be injected too (`requestFactory:`, `streamFactory:`). When you inject a client, set its timeout yourself.

## Laravel

No service provider needed: bind the client in `app/Providers/AppServiceProvider.php`.

```php
use XDive\Client;

public function register(): void
{
    $this->app->singleton(Client::class, fn () => new Client(
        apiKey: config('services.xdive.key'),
        baseUrl: config('services.xdive.base_url'),   // null falls back to the default
        orgId: config('services.xdive.org_id'),
    ));
}
```

```php
// config/services.php
'xdive' => [
    'key' => env('XDIVE_API_KEY'),
    'base_url' => env('XDIVE_BASE_URL'),
    'org_id' => env('XDIVE_ORG_ID'),
],
```

Then type-hint `XDive\Client` in controllers, jobs and commands. To use Laravel's HTTP stack settings, pass an injected Guzzle client via `httpClient:`.

## Symfony

```yaml
# config/services.yaml
services:
    XDive\Client:
        arguments:
            $apiKey: '%env(XDIVE_API_KEY)%'
            $baseUrl: '%env(default::XDIVE_BASE_URL)%'
            $orgId: '%env(default::XDIVE_ORG_ID)%'
            $httpClient: '@psr18.http_client'   # optional: reuse Symfony's PSR-18 client
```

`default::VAR` resolves to `null` when the variable is unset, which makes the client use its defaults. `@psr18.http_client` requires `symfony/http-client` and `nyholm/psr7`; leave `$httpClient` out to use Guzzle.

## Error handling

Every exception the SDK throws implements the marker interface `XDive\Exception\XDiveException`.

```php
use XDive\Exception\ApiException;
use XDive\Exception\CrawlTimeoutException;
use XDive\Exception\QuotaExceededException;
use XDive\Exception\RateLimitedException;
use XDive\Exception\XDiveException;

try {
    $client->scrape('https://example.com');
} catch (RateLimitedException $e) {          // retries ran out on 429 / 503 SERVER_BUSY / 409 IDEMPOTENCY_KEY_IN_PROGRESS
    sleep((int) ceil($e->retryAfter()));
} catch (QuotaExceededException $e) {        // HTTP 402: plan quota exhausted
    // upgrade the plan
} catch (ApiException $e) {                  // any other error response
    error_log(sprintf('%d %s (%s) request=%s docs=%s',
        $e->status(), $e->errorCode(), $e->getMessage(), $e->requestId(), $e->docsUrl()));
    if ($e->retryable()) { /* the server says the same request may succeed later */ }
} catch (XDiveException $e) {                // TransportException, CrawlTimeoutException, ConfigurationException ...
}
```

`ApiException extends \RuntimeException` and exposes `status()`, `errorCode()` (null for older error bodies), `retryable()`, `requestId()` (the `X-Request-ID` header) and `docsUrl()`. `RateLimitedException` adds `retryAfter()` in float seconds; `CrawlTimeoutException` (also an `ApiException`, status 408) adds `crawlId()`. `TransportException` means no HTTP response arrived after all retries; the PSR-18 exception is `getPrevious()`.

## Retries and idempotency

The client retries, up to `maxRetries` times (default 3), on:

- `429`, `503` with code `SERVER_BUSY`, and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS`, waiting the `Retry-After` seconds (default 1, fractions allowed);
- transport errors and timeouts for GET and POST, with backoff 1s, 2s, 4s, capped at 8s.

Every other status is never retried. Every POST carries an `Idempotency-Key` header (a random UUID v4) generated once per call and reused by every retry, so a retried scrape or crawl never runs twice. Pass your own to make a call safe to repeat from your side too:

```php
$client->startCrawl('https://example.com', idempotencyKey: 'nightly-2026-10-18');
```

Calls with the same key return the first call's response. A new call without a key gets a new key.

## Testing your code

`Client` accepts any PSR-18 client, so tests can inject a mock (for example Guzzle's `MockHandler`). The constructor also takes `sleeper:` and `clock:` callables to make retries and `waitForCrawl()` run without real delays.

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-php · All docs for agents: https://xdive.dev/llms.txt