Crawl

Start at one URL, follow its links, and convert every page reached. A crawl is asynchronous: the request returns a crawl id at once, a worker does the fetching, and you poll the crawl or receive a webhook when it finishes. Use it to ingest a documentation site, a blog or a section of a site; use Map when you only need the URLs.

Requires an API key with the crawl scope. Starting a crawl counts against your plan's request rate limit; polling it does not.

# Start the crawl
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/crawl" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "max_pages": 50,
    "include_paths": ["^/guides/"],
    "formats": ["markdown"]
  }'
# {"id":"crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d","status":"queued"}
 
# Poll until status is success, error or cancelled
curl "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY"

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
 
# crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
result = xdive.crawl(
    "https://docs.example.com",
    max_depth=2,
    max_pages=50,
    include_paths=["^/guides/"],
    formats=["markdown"],
)
for page in result["results"]:
    print(page["url"], page.get("error", ""))
 
# Or start it and poll yourself:
crawl_id = xdive.start_crawl("https://docs.example.com", max_pages=50)
record = xdive.wait_for_crawl(crawl_id, timeout=600)

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
 
// crawl() starts the crawl and polls until it finishes (ETag and Retry-After aware).
const result = await xdive.crawl("https://docs.example.com", {
  max_depth: 2,
  max_pages: 50,
  include_paths: ["^/guides/"],
  formats: ["markdown"],
});
for (const page of result.results ?? []) console.log(page.url, page.error ?? "");
 
// Or start it and poll yourself:
const crawlId = await xdive.startCrawl("https://docs.example.com", { max_pages: 50 });
const record = await xdive.waitForCrawl(crawlId, { timeoutMs: 600_000 });

Parameters

Structured records

extract_schema maps field names to CSS selectors. With container, each element it matches yields one record and the field selectors are evaluated inside it, which suits lists of products or posts; without it, each page yields one record. A field takes the element's text, or an attribute's value when attr is set. Invalid selectors are skipped.

{
  "extract_schema": {
    "container": "article.post",
    "fields": [
      { "name": "title", "selector": "h2" },
      { "name": "link", "selector": "a", "attr": "href" }
    ]
  }
}

llm_extract describes fields in words instead: fields is a list of name, description and an optional type hint (string, number or boolean); prompt adds guidance and multiple: true asks for one record per item on list pages. It runs on the first 50 successfully fetched pages of a crawl. Both put their output in each page's records, a list of objects with string values.

Response

The crawl is queued and the response is 202 Accepted:

{ "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d", "status": "queued" }

Get a crawl

Returns the crawl's status, and its pages once it has finished. Batch scrapes use the same endpoint.

{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "maxDepth": 2,
  "durationMs": 18342,
  "httpPages": 37,
  "cachePages": 4,
  "impersonatedPages": 1,
  "browserPages": 0,
  "fetchAttempts": 39,
  "fetchCostUnits": 40,
  "error": "",
  "createdAt": "2026-10-15T09:30:12.481557Z",
  "results": [
    {
      "url": "https://docs.example.com/guides/install",
      "title": "Install",
      "description": "Install the CLI on macOS, Linux and Windows.",
      "depth": 1,
      "status_code": 200,
      "word_count": 618,
      "fetched_at": "2026-10-15T09:30:16.020114Z",
      "fetch_route": "http",
      "fetch_attempts": 1,
      "fetch_cost_units": 1,
      "markdown": "# Install\n\nDownload the latest release..."
    }
  ]
}

Each page has the fields of the scrape page object except the links list, plus:

Status values

success, error and cancelled are final.

Polling with ETag and Retry-After

Every response carries an ETag that changes only when the crawl's status, page count or duration does. Send it back in If-None-Match and an unchanged crawl answers 304 Not Modified with no body. While a crawl is queued or running, responses also carry Retry-After: 2, the suggested seconds between polls. The SDKs' crawl, wait_for_crawl (Python) and waitForCrawl (JavaScript) do both for you.

curl -i "https://xdive.dev/api/orgs/$ORG_ID/crawls/crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d" \
  -H "X-API-Key: $XDIVE_API_KEY" \
  -H 'If-None-Match: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"'
# HTTP/1.1 304 Not Modified
# Etag: "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d.running.0.0"
# Retry-After: 2

Polling, cancelling and listing errors are not counted against your plan's request rate limit.

Cancel a crawl

Cancels a queued or running crawl or batch scrape and returns {"id": "crl_…", "status": "cancelled"}. A running crawl notices within about two seconds and saves the pages fetched so far; a crawl cancelled before a worker picked it up ends with no results and sends no webhook. Cancelling a finished crawl returns 409.

SDK: xdive.cancel_crawl(crawl_id) in Python, xdive.cancelCrawl(crawlId) in JavaScript.

List failed pages

Returns only the pages that failed, so you can audit a large crawl without downloading every page. Pages refused by robots.txt are listed separately.

{
  "id": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "status": "success",
  "error": "",
  "errors": [
    { "url": "https://docs.example.com/guides/legacy", "error": "unexpected status 404" }
  ],
  "robots_blocked": ["https://docs.example.com/guides/internal"]
}

SDK: xdive.crawl_errors(crawl_id) in Python, xdive.crawlErrors(crawlId) in JavaScript.

Path filters

include_paths and exclude_paths are RE2 regular expressions matched against the path of each discovered link (/guides/install, without host or query). They are not anchored: guides matches anywhere in the path, ^/guides/ only at the start. An exclude match always wins over an include match. The filters decide which links are followed; the seed URL is fetched regardless. Invalid patterns are rejected with 400 before the crawl is queued.

Webhooks

Set webhook_url to have the crawl POST its progress to you. The URL must resolve to a public address (400 with WEBHOOK_TARGET_DENIED otherwise). By default the platform requires a webhook_secret; a request without one is rejected with 400 and SIGNED_WEBHOOK_REQUIRED.

Events

Without webhook_events you receive completed and failed. The delivered event name is in the X-Crawler-Event header and the payload's event field; for crawl.finished, read status to tell the outcomes apart.

Payloads

{
  "event": "crawl.finished",
  "crawlId": "crl_7d1e4b9a2c6f3e8d0a5b7c9e1f3a5c7d",
  "orgId": "org_5f2c9a1e7b3d4c6f8a0b2e4d6c8f1a3b",
  "url": "https://docs.example.com",
  "status": "success",
  "pages": 42,
  "durationMs": 18342,
  "finishedAt": "2026-10-15T09:30:30.823910Z"
}

crawl.finished adds error when the crawl failed. crawl.started carries event, crawlId, orgId, url and sentAt. crawl.page adds a page object with the page's url, title, error and depth; fetch the content from the crawl endpoint.

Verifying signatures

When the crawl has a webhook_secret, every delivery carries three headers:

To verify, compute the HMAC over the exact bytes you received (before any JSON parsing), compare it with the header in constant time, and reject timestamps outside a window you choose, such as five minutes.

import hashlib
import hmac
import time
 
def verify_webhook(body: bytes, headers, secret: str, tolerance: int = 300) -> bool:
    timestamp = headers.get("X-Crawler-Timestamp", "")
    signature = headers.get("X-Crawler-Signature-V1", "").removeprefix("v1=")
    if not timestamp.isdigit() or abs(time.time() - int(timestamp)) > tolerance:
        return False
    expected = hmac.new(secret.encode(), timestamp.encode() + b"." + body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, signature)

import { createHmac, timingSafeEqual } from "node:crypto";
 
export function verifyWebhook(body: Buffer, headers: Record<string, string | undefined>, secret: string, toleranceSec = 300): boolean {
  const timestamp = headers["x-crawler-timestamp"] ?? "";
  const signature = (headers["x-crawler-signature-v1"] ?? "").replace(/^v1=/, "");
  if (!/^\d+$/.test(timestamp) || Math.abs(Date.now() / 1000 - Number(timestamp)) > toleranceSec) return false;
  const expected = createHmac("sha256", secret).update(`${timestamp}.`).update(body).digest("hex");
  return expected.length === signature.length && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}

Delivery and retries

A delivery succeeds when your endpoint answers with a 2xx status within 10 seconds. crawl.finished is attempted up to three times, one and then two seconds apart; crawl.started and crawl.page are attempted once. Delivery is best effort, so treat the crawl endpoint as the source of truth.

Limits

• At most 500 pages per crawl, and 2 link hops by default.
• A crawl runs for at most 2 minutes. Pages fetched by then are kept, and the crawl succeeds if any of them did.
• Each page is read up to 5 MB.
• llm_extract runs on at most 50 pages per crawl.
• At most 10 custom headers. The request body is limited to 1 MB.

Starting a crawl counts one crawl against your plan; the pages in the result, including pages that failed, are counted against your monthly page quota when it finishes. If the page quota is already used up, the crawl is refused with 402. See Usage and billing.

Errors

See Errors and limits for authentication errors and retries.

Related

• Batch scrape for a list of known URLs.
• Map to list a site's URLs without fetching content.
• Scrape for formats and fetch telemetry.