PHP SDK

The official PHP SDK for the XDive web-data API: scrape, crawl, search and research the web as clean Markdown. PHP 8.1+, works in plain PHP, Laravel and Symfony. HTTP goes through PSR-18, so you can plug in any client; Guzzle is the default.

Install

bash

composer require xdive/xdive

Quickstart

php

<?php
 
use XDive\Client;
use XDive\Format;
 
require 'vendor/autoload.php';
 
$client = new Client(apiKey: getenv('XDIVE_API_KEY'));   // or Client::fromEnvironment()
 
// Scrape one page
$page = $client->scrape('https://example.com', [Format::MARKDOWN]);
echo $page->title, "\n", $page->markdown(), "\n";
 
// Crawl a site and wait for the result
$crawl = $client->crawl('https://example.com', [Format::MARKDOWN], ['limit' => 20, 'max_depth' => 2]);
foreach ($crawl->results as $p) {
    echo $p->url, ' ', strlen($p->markdown()), " chars\n";
}
 
// Search the web
$search = $client->search('php 8.4 release notes', limit: 5, includeAnswer: true);
foreach ($search->results as $hit) {
    echo "{$hit->rank}. {$hit->title} {$hit->url}\n";
}
 
// Research a question with verified citations
$research = $client->research('How does HTTP/3 differ from HTTP/2?', depth: 'quick');
echo $research->answer, "\n";
foreach ($research->citations as $c) {
    echo "[{$c->number}] {$c->url}\n";
}

Methods

Method	What it does
`scrape($url, $formats, $extra)`	One page, synchronously. Returns `Page`.
`startCrawl` / `crawl` / `getCrawl` / `waitForCrawl` / `cancelCrawl` / `crawlErrors`	The crawl lifecycle. `crawl()` is start + wait.
`startBatchScrape` / `batchScrape`	Up to 100 URLs through the job pipeline (follow with `waitForCrawl`).
`map($url, search:, limit:, maxDepth:)`	Discover a site's URLs. Returns `list<string>`.
`search($query, ...)`	Web search. Returns `SearchResponse`.
`research($query, depth:, output:, schema:)`	Cited answer. Returns `ResearchResult`.
`extract($urls, prompt:, schema:)`	Structured data across up to 10 URLs.
`llmsTxt($url, maxUrls:, fullText:)`	Generate `llms.txt` for a site.
`whoami()` / `orgId()`	The key's organization, scopes and plan.
`Client::pageMarkdown($page)` / `$page->markdown()`	Markdown of a page, whichever field carries it.

All methods are synchronous and return readonly value objects (Page, CrawlRecord, SearchResponse, ResearchResult, Citation, ...) with public readonly properties for the stable fields. Every value object has raw(), the decoded array exactly as received, for fields the SDK does not model. Open-ended parts (extract(), llmsTxt(), whoami()) are plain arrays.

Options

Common options are named arguments; anything else the API accepts goes in the $extra array (it wins over named options, null values are dropped):

php

$page = $client->scrape(
    'https://example.com/pricing',
    formats: [
        Format::MARKDOWN,                                     // plain format
        Format::object(Format::JSON, prompt: 'Extract plans', // object format
            schema: ['type' => 'object', 'properties' => ['plans' => ['type' => 'array']]]),
    ],
    extra: ['only_main_content' => true, 'wait_for' => 1000],
);

Format::names('markdown', 'links') builds a list of plain formats; Format::object($type, prompt:, schema:, question:, query:, maxResults:, extra:) builds an object format.

Waiting for crawls

waitForCrawl() polls with If-None-Match, so an unchanged crawl costs a bodiless 304, and follows the server's Retry-After. It throws CrawlTimeoutException after $timeout seconds (default 600); the crawl keeps running on the server.

php

$id = $client->startCrawl('https://example.com', extra: ['limit' => 100]);
$record = $client->waitForCrawl($id, timeout: 300.0, pollInterval: 2.0);
$failed = $client->crawlErrors($id);   // failed pages and robots.txt-blocked URLs

Configuration

php

$client = new Client(
    apiKey: 'xd_...',                          // default: XDIVE_API_KEY, then CRAWLER_API_KEY
    baseUrl: Client::DEFAULT_BASE_URL,         // default: XDIVE_BASE_URL, then CRAWLER_BASE_URL
    orgId: 'org_123',                          // skips the one-time GET /whoami discovery
    maxRetries: 3,
    timeout: 120.0,                            // per attempt, default Guzzle client only
    httpClient: $psr18Client,                  // any Psr\Http\Client\ClientInterface
);

Variable	Purpose
`XDIVE_API_KEY` (fallback `CRAWLER_API_KEY`)	API key
`XDIVE_BASE_URL` (fallback `CRAWLER_BASE_URL`)	API origin including `/api`

The organization id is discovered once with GET /whoami and cached on the client instance. The client holds no other state, so create it once and reuse it.

To use another HTTP client, pass any PSR-18 client; request and stream factories default to guzzlehttp/psr7 and can be injected too (requestFactory:, streamFactory:). When you inject a client, set its timeout yourself.

Laravel

No service provider needed: bind the client in app/Providers/AppServiceProvider.php.

php

use XDive\Client;
 
public function register(): void
{
    $this->app->singleton(Client::class, fn () => new Client(
        apiKey: config('services.xdive.key'),
        baseUrl: config('services.xdive.base_url'),   // null falls back to the default
        orgId: config('services.xdive.org_id'),
    ));
}

php

// config/services.php
'xdive' => [
    'key' => env('XDIVE_API_KEY'),
    'base_url' => env('XDIVE_BASE_URL'),
    'org_id' => env('XDIVE_ORG_ID'),
],

Then type-hint XDive\Client in controllers, jobs and commands. To use Laravel's HTTP stack settings, pass an injected Guzzle client via httpClient:.

Symfony

yaml

# config/services.yaml
services:
    XDive\Client:
        arguments:
            $apiKey: '%env(XDIVE_API_KEY)%'
            $baseUrl: '%env(default::XDIVE_BASE_URL)%'
            $orgId: '%env(default::XDIVE_ORG_ID)%'
            $httpClient: '@psr18.http_client'   # optional: reuse Symfony's PSR-18 client

default::VAR resolves to null when the variable is unset, which makes the client use its defaults. @psr18.http_client requires symfony/http-client and nyholm/psr7; leave $httpClient out to use Guzzle.

Error handling

Every exception the SDK throws implements the marker interface XDive\Exception\XDiveException.

php

use XDive\Exception\ApiException;
use XDive\Exception\CrawlTimeoutException;
use XDive\Exception\QuotaExceededException;
use XDive\Exception\RateLimitedException;
use XDive\Exception\XDiveException;
 
try {
    $client->scrape('https://example.com');
} catch (RateLimitedException $e) {          // retries ran out on 429 / 503 SERVER_BUSY / 409 IDEMPOTENCY_KEY_IN_PROGRESS
    sleep((int) ceil($e->retryAfter()));
} catch (QuotaExceededException $e) {        // HTTP 402: plan quota exhausted
    // upgrade the plan
} catch (ApiException $e) {                  // any other error response
    error_log(sprintf('%d %s (%s) request=%s docs=%s',
        $e->status(), $e->errorCode(), $e->getMessage(), $e->requestId(), $e->docsUrl()));
    if ($e->retryable()) { /* the server says the same request may succeed later */ }
} catch (XDiveException $e) {                // TransportException, CrawlTimeoutException, ConfigurationException ...
}

ApiException extends \RuntimeException and exposes status(), errorCode() (null for older error bodies), retryable(), requestId() (the X-Request-ID header) and docsUrl(). RateLimitedException adds retryAfter() in float seconds; CrawlTimeoutException (also an ApiException, status 408) adds crawlId(). TransportException means no HTTP response arrived after all retries; the PSR-18 exception is getPrevious().

Retries and idempotency

The client retries, up to maxRetries times (default 3), on:

429, 503 with code SERVER_BUSY, and 409 with code IDEMPOTENCY_KEY_IN_PROGRESS, waiting the Retry-After seconds (default 1, fractions allowed);
transport errors and timeouts for GET and POST, with backoff 1s, 2s, 4s, capped at 8s.

Every other status is never retried. Every POST carries an Idempotency-Key header (a random UUID v4) generated once per call and reused by every retry, so a retried scrape or crawl never runs twice. Pass your own to make a call safe to repeat from your side too:

php

$client->startCrawl('https://example.com', idempotencyKey: 'nightly-2026-10-18');

Calls with the same key return the first call's response. A new call without a key gets a new key.

Testing your code

Client accepts any PSR-18 client, so tests can inject a mock (for example Guzzle's MockHandler). The constructor also takes sleeper: and clock: callables to make retries and waitForCrawl() run without real delays.