# Crawl

Start at one URL, follow its links, and convert every page reached. A crawl is asynchronous: the request returns a crawl id at once, a worker does the fetching, and you poll the crawl or receive a webhook when it finishes. Use it to ingest a documentation site, a blog or a section of a site; use [Map](https://xdive.dev/docs/map.md) when you only need the URLs.

`POST https://xdive.dev/api/orgs/{id}/crawl`

Requires an API key with the `crawl` scope. Starting a crawl counts against your plan's request rate limit; polling it does not.

**cURL**

```bash
# Start the crawl
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "max_pages": 50,
    "include_paths": ["^/guides/"],
    "formats": ["markdown"]
  }'
# {"id":"crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d","status":"queued"}

# Poll until status is success, error or cancelled
curl "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY"
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY

# crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
result = xdive.crawl(
    "https://docs.example.com",
    max_depth=2,
    max_pages=50,
    include_paths=["^/guides/"],
    formats=["markdown"],
)
for page in result["results"]:
    print(page["url"], page.get("error", ""))

# Or start it and poll yourself:
crawl_id = xdive.start_crawl("https://docs.example.com", max_pages=50)
record = xdive.wait_for_crawl(crawl_id, timeout=600)
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY

// crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
const result = await xdive.crawl("https://docs.example.com", {
  max_depth: 2,
  max_pages: 50,
  include_paths: ["^/guides/"],
  formats: ["markdown"],
});
for (const page of result.results ?? []) console.log(page.url, page.error ?? "");

// Or start it and poll yourself:
const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 50 });
const record = await xdive.waitForCrawl(crawlId, { timeoutMs: 600_000 });
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The seed URL. It is always fetched. |
| `max_depth` | integer | `2` | Link hops to follow from the seed. `0` fetches only the seed. |
| `max_pages` | integer | `100` | Pages to fetch, at most 500. `0` or a value above 500 means 500. |
| `same_domain` | boolean | `true` | Follow only links on the seed's registrable domain: for `https://docs.example.com`, links to `www.example.com` are followed, links to `example.org` are not. |
| `include_paths` | array of strings | none | RE2 patterns. When set, a discovered link is followed only if its URL path matches at least one. See [Path filters](#path-filters). |
| `exclude_paths` | array of strings | none | RE2 patterns. A discovered link whose path matches any of them is skipped. |
| `formats` | array | none | Representations for each page, as on [Scrape](https://xdive.dev/docs/scrape.md#formats). `screenshot` and `audio` are not available on crawls. Without `formats`, each page has a `markdown` field. |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip URLs that robots.txt disallows. They are reported by the [errors endpoint](#list-failed-pages). |
| `adaptive_fetch` | boolean | `true` | Plain HTTP first; on a block or bot filter, retry with a Chrome TLS fingerprint, and on a challenge page or JavaScript shell, render in a headless browser. `false` uses plain HTTP only. Cannot be combined with `impersonate` or `render`. |
| `impersonate` | boolean | `false` | Fetch every page with a Chrome TLS fingerprint and matching User-Agent. Turns `adaptive_fetch` off. |
| `render` | boolean | `false` | Render every page in a headless browser so JavaScript runs. Turns `adaptive_fetch` off. |
| `bypass_cache` | boolean | `false` | Neither read nor write the page cache. |
| `max_age` | integer | `0` | Cache freshness window in milliseconds, as on [Scrape](https://xdive.dev/docs/scrape.md#caching-and-max_age). `0` uses the server default. |
| `headers` | object | none | Extra request headers for every page fetch, with the same rules as [Scrape](https://xdive.dev/docs/scrape.md#custom-headers). |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses in every page with typed tokens. |
| `extract_schema` | object | none | Pull records from each page with CSS selectors. See [Structured records](#structured-records). |
| `llm_extract` | object | none | Pull records from each page with an LLM. Requires an LLM. |
| `webhook_url` | string | none | A public URL that receives a signed POST for the events in `webhook_events`. See [Webhooks](#webhooks). |
| `webhook_secret` | string | none | Key for the HMAC-SHA256 signature. Required unless the platform's signed-webhook policy is off. |
| `webhook_events` | array of strings | `["completed", "failed"]` | Any of `started`, `page`, `completed`, `failed`. |

### Structured records

`extract_schema` maps field names to CSS selectors. With `container`, each element it matches yields one record and the field selectors are evaluated inside it, which suits lists of products or posts; without it, each page yields one record. A field takes the element's text, or an attribute's value when `attr` is set. Invalid selectors are skipped.

```json
{
  "extract_schema": {
    "container": "article.post",
    "fields": [
      { "name": "title", "selector": "h2" },
      { "name": "link", "selector": "a", "attr": "href" }
    ]
  }
}
```

`llm_extract` describes fields in words instead: `fields` is a list of `name`, `description` and an optional `type` hint (`string`, `number` or `boolean`); `prompt` adds guidance and `multiple: true` asks for one record per item on list pages. It runs on the first 50 successfully fetched pages of a crawl. Both put their output in each page's `records`, a list of objects with string values.

## Response

The crawl is queued and the response is `202 Accepted`:

```json
{ "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d", "status": "queued" }
```

## Get a crawl

`GET https://xdive.dev/api/orgs/{id}/crawls/{crawlId}`

Returns the crawl's status, and its pages once it has finished. Batch scrapes use the same endpoint.

```json
{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "maxDepth": 2,
  "durationMs": 18342,
  "httpPages": 37,
  "cachePages": 4,
  "impersonatedPages": 1,
  "browserPages": 0,
  "fetchAttempts": 39,
  "fetchCostUnits": 40,
  "error": "",
  "createdAt": "2026-10-15T09:30:12.481557Z",
  "results": [
    {
      "url": "https://docs.example.com/guides/install",
      "title": "Install",
      "description": "Install the CLI on macOS, Linux and Windows.",
      "depth": 1,
      "status_code": 200,
      "word_count": 618,
      "fetched_at": "2026-10-15T09:30:16.020114Z",
      "fetch_route": "http",
      "fetch_attempts": 1,
      "fetch_cost_units": 1,
      "markdown": "# Install\n\nDownload the latest release..."
    }
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `id` | string | The crawl id. |
| `url` | string | The seed URL. |
| `status` | string | See [Status values](#status-values). |
| `pages` | integer | Pages in the result, including pages that failed. Set when the crawl finishes. |
| `maxDepth` | integer | The depth the crawl ran with. |
| `durationMs` | integer | Time the crawl took. |
| `httpPages`, `cachePages`, `impersonatedPages`, `browserPages` | integer | Pages by [fetch route](https://xdive.dev/docs/scrape.md#fetch-telemetry). |
| `fetchAttempts`, `fetchCostUnits` | integer | Sums of the pages' fetch attempts and cost units. |
| `error` | string | Why the crawl failed; empty otherwise. |
| `createdAt` | string | When the crawl was submitted. |
| `results` | array or null | The pages, in no particular order, once the crawl has finished; `null` before. |

Each page has the fields of the [scrape page object](https://xdive.dev/docs/scrape.md#the-page-object) except the `links` list, plus:

| Field | Type | Description |
|---|---|---|
| `depth` | integer | Link hops from the seed. |
| `markdown` | string | The page's Markdown, when the crawl had no `formats`. |
| `artifacts` | object | The requested formats, when the crawl had `formats`. |
| `records` | array | Output of `extract_schema` or `llm_extract`. |
| `error` | string | Why this page failed. |
| `error_code` | string | For sites that block data-centre networks: `PROXY_NOT_ENABLED` when your organization has no residential proxy access (or has used its monthly allowance), `SITE_RESTRICTED` when the proxy provider refused the destination. See [Government and banking sites](https://xdive.dev/docs/scrape.md#government-and-banking-sites). Otherwise absent. |
| `policy_refusal` | boolean | `true` when robots.txt disallowed the page. |

### Status values

| Status | Meaning |
|---|---|
| `queued` | Accepted and waiting for a worker. |
| `running` | A worker is fetching pages. |
| `success` | Finished, and at least one page was fetched without error. |
| `error` | Finished without a single successful page. `error` holds the first page's error, or `no pages crawled`. |
| `cancelled` | Cancelled with `DELETE`. `results` holds the pages fetched before the cancellation. |

`success`, `error` and `cancelled` are final.

### Polling with ETag and Retry-After

Every response carries an `ETag` that changes only when the crawl's status, page count or duration does. Send it back in `If-None-Match` and an unchanged crawl answers `304 Not Modified` with no body. While a crawl is `queued` or `running`, responses also carry `Retry-After: 2`, the suggested seconds between polls. The SDKs' `crawl`, `wait_for_crawl` (Python) and `waitForCrawl` (JavaScript) do both for you.

```bash
curl -i "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY" \
  -H 'If-None-Match: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"'
# HTTP/1.1 304 Not Modified
# Etag: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"
# Retry-After: 2
```

Polling, cancelling and listing errors are not counted against your plan's request rate limit.

## Cancel a crawl

`DELETE https://xdive.dev/api/orgs/{id}/crawls/{crawlId}`

Cancels a queued or running crawl or batch scrape and returns `{"id": "crl_…", "status": "cancelled"}`. A running crawl notices within about two seconds and saves the pages fetched so far; a crawl cancelled before a worker picked it up ends with no results and sends no webhook. Cancelling a finished crawl returns `409`.

SDK: `xdive.cancel_crawl(crawl_id)` in Python, `xdive.cancelCrawl(crawlId)` in JavaScript.

## List failed pages

`GET https://xdive.dev/api/orgs/{id}/crawls/{crawlId}/errors`

Returns only the pages that failed, so you can audit a large crawl without downloading every page. Pages refused by robots.txt are listed separately.

```json
{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "status": "success",
  "error": "",
  "errors": [
    { "url": "https://docs.example.com/guides/legacy", "error": "unexpected status 404" }
  ],
  "robots_blocked": ["https://docs.example.com/guides/internal"]
}
```

SDK: `xdive.crawl_errors(crawl_id)` in Python, `xdive.crawlErrors(crawlId)` in JavaScript.

## Path filters

`include_paths` and `exclude_paths` are [RE2](https://github.com/google/re2/wiki/Syntax) regular expressions matched against the path of each discovered link (`/guides/install`, without host or query). They are not anchored: `guides` matches anywhere in the path, `^/guides/` only at the start. An exclude match always wins over an include match. The filters decide which links are followed; the seed URL is fetched regardless. Invalid patterns are rejected with `400` before the crawl is queued.

## Webhooks

Set `webhook_url` to have the crawl POST its progress to you. The URL must resolve to a public address (`400` with `WEBHOOK_TARGET_DENIED` otherwise). By default the platform requires a `webhook_secret`; a request without one is rejected with `400` and `SIGNED_WEBHOOK_REQUIRED`.

### Events

| Subscription in `webhook_events` | Delivered event | When |
|---|---|---|
| `started` | `crawl.started` | A worker starts the crawl. |
| `page` | `crawl.page` | Each page is fetched, whether it succeeded or failed. |
| `completed` | `crawl.finished` | The crawl ends with status `success`. |
| `failed` | `crawl.finished` | The crawl ends with status `error` or `cancelled`. |

Without `webhook_events` you receive `completed` and `failed`. The delivered event name is in the `X-Crawler-Event` header and the payload's `event` field; for `crawl.finished`, read `status` to tell the outcomes apart.

### Payloads

```json
{
  "event": "crawl.finished",
  "crawlId": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "orgId": "org_5f2c9a1e7b3d4c6f8a0b2e4d6c8f1a3b",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "durationMs": 18342,
  "finishedAt": "2026-10-15T09:30:30.823910Z"
}
```

`crawl.finished` adds `error` when the crawl failed. `crawl.started` carries `event`, `crawlId`, `orgId`, `url` and `sentAt`. `crawl.page` adds a `page` object with the page's `url`, `title`, `error` and `depth`; fetch the content from the crawl endpoint.

### Verifying signatures

When the crawl has a `webhook_secret`, every delivery carries three headers:

| Header | Value |
|---|---|
| `X-Crawler-Timestamp` | Unix time in seconds when the delivery was signed. |
| `X-Crawler-Signature-V1` | `v1=` followed by the hex HMAC-SHA256 of the timestamp, a `.`, and the raw request body, keyed with your secret. |
| `X-Crawler-Signature` | The hex HMAC-SHA256 of the raw body alone. Kept for older receivers; prefer V1, which lets you reject replays. |

To verify, compute the HMAC over the exact bytes you received (before any JSON parsing), compare it with the header in constant time, and reject timestamps outside a window you choose, such as five minutes.

**Python**

```python
import hashlib
import hmac
import time

def verify_webhook(body: bytes, headers, secret: str, tolerance: int = 300) -> bool:
    timestamp = headers.get("X-Crawler-Timestamp", "")
    signature = headers.get("X-Crawler-Signature-V1", "").removeprefix("v1=")
    if not timestamp.isdigit() or abs(time.time() - int(timestamp)) > tolerance:
        return False
    expected = hmac.new(secret.encode(), timestamp.encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)
```

**JavaScript**

```ts
import { createHmac, timingSafeEqual } from "node:crypto";

export function verifyWebhook(body: Buffer, headers: Record<string, string | undefined>, secret: string, toleranceSec = 300): boolean {
  const timestamp = headers["x-crawler-timestamp"] ?? "";
  const signature = (headers["x-crawler-signature-v1"] ?? "").replace(/^v1=/, "");
  if (!/^\d+$/.test(timestamp) || Math.abs(Date.now() / 1000 - Number(timestamp)) > toleranceSec) return false;
  const expected = createHmac("sha256", secret).update(`${timestamp}.`).update(body).digest("hex");
  return expected.length === signature.length && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}
```

### Delivery and retries

A delivery succeeds when your endpoint answers with a 2xx status within 10 seconds. `crawl.finished` is attempted up to three times, one and then two seconds apart; `crawl.started` and `crawl.page` are attempted once. Delivery is best effort, so treat the crawl endpoint as the source of truth.

## Limits

- At most 500 pages per crawl, and 2 link hops by default.
- A crawl runs for at most 2 minutes. Pages fetched by then are kept, and the crawl succeeds if any of them did.
- Each page is read up to 5 MB.
- `llm_extract` runs on at most 50 pages per crawl.
- At most 10 custom headers. The request body is limited to 1 MB.

Starting a crawl counts one crawl against your plan; the pages in the result, including pages that failed, are counted against your monthly page quota when it finishes. If the page quota is already used up, the crawl is refused with `402`. See [Usage and billing](https://xdive.dev/docs/usage.md).

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | Missing `url`, an invalid format or one not allowed on crawls, an invalid path pattern, header or `webhook_events` entry, a negative `max_age`, or `adaptive_fetch` combined with `impersonate` or `render`. |
| `400` | `WEBHOOK_TARGET_DENIED` | `webhook_url` is not a public URL. |
| `400` | `SIGNED_WEBHOOK_REQUIRED` | `webhook_url` was set without `webhook_secret`. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key lacks the `crawl` scope. |
| `404` | | No crawl with this id in your organization (status, cancel and errors). |
| `409` | | Cancelling a crawl that has already finished. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `501` | | `llm_extract` or an LLM format was requested and no LLM is configured. |
| `503` | | The crawl could not be queued. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) for a list of known URLs.
- [Map](https://xdive.dev/docs/map.md) to list a site's URLs without fetching content.
- [Scrape](https://xdive.dev/docs/scrape.md) for formats and fetch telemetry.

---

Source: https://xdive.dev/docs/crawl · All docs for agents: https://xdive.dev/llms.txt