Batch scrape

Scrape a list of up to 100 known URLs as one asynchronous job. Each URL is fetched on its own, without following links, through the same pipeline as a crawl. The job has a crawl id, so you follow it, cancel it and list its failures with the crawl endpoints, and it can notify a webhook.

POST/orgs/{id}/batch/scrape

Requires an API key with the scrape or the crawl scope. Submitting a batch counts against your plan's request rate limit.

bash

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/batch/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/pricing",
      "https://example.com/about",
      "https://example.com/blog/launch"
    ],
    "formats": ["markdown", "links"],
    "ignore_invalid_urls": true
  }'
# {"id":"crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f","status":"queued","urls":3,"invalid_urls":[]}

python

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
record = xdive.batch_scrape(
    [
        "https://example.com/pricing",
        "https://example.com/about",
        "https://example.com/blog/launch",
    ],
    formats=["markdown", "links"],
    ignore_invalid_urls=True,
)
for page in record["results"]:
    print(page["url"], page.get("error", ""))
 
# wait=False returns the job id instead of waiting:
job_id = xdive.batch_scrape(["https://example.com/pricing"], wait=False)

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const record = await xdive.batchScrape(
  [
    "https://example.com/pricing",
    "https://example.com/about",
    "https://example.com/blog/launch",
  ],
  { formats: ["markdown", "links"], ignore_invalid_urls: true },
);
for (const page of record.results ?? []) console.log(page.url, page.error ?? "");

The SDK methods submit the batch and poll it until it finishes (wait=False in Python returns the id instead). With cURL, poll GET /orgs/{id}/crawls/{crawlId} as described on Crawl.

Parameters

Field	Type	Default	Description
`urls`	array of strings	required	1 to 100 absolute `http` or `https` URLs. Blank entries are skipped and exact duplicates are scraped once.
`ignore_invalid_urls`	boolean	`false`	Skip invalid entries instead of rejecting the request. Skipped entries are returned in `invalid_urls`.
`formats`	array	none	Representations for each page, as on Scrape. `screenshot` and `audio` are not available. Without `formats`, each page has a `markdown` field.
`extract`	boolean	`true`	Main-content extraction: drop navigation, headers and footers.
`respect_robots`	boolean	`true`	Skip URLs that robots.txt disallows.
`adaptive_fetch`	boolean	`true`	Plain HTTP first, escalating to a Chrome TLS fingerprint and then a headless browser for blocked pages, challenges and JavaScript shells. `false` uses plain HTTP only.
`bypass_cache`	boolean	`false`	Neither read nor write the page cache.
`max_age`	integer	`0`	Cache freshness window in milliseconds, as on Scrape.
`headers`	object	none	Extra request headers for every fetch, with the rules on Scrape.
`redact_pii`	boolean	`false`	Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses with typed tokens.
`webhook_url`	string	none	Receives signed event POSTs, as for crawls.
`webhook_secret`	string	none	HMAC-SHA256 signing key. Required unless the platform's signed-webhook policy is off.
`webhook_events`	array of strings	`["completed", "failed"]`	Any of `started`, `page`, `completed`, `failed`.

A URL is valid when it parses as an absolute URL with an http or https scheme and a host.

Response

The batch is queued and the response is 202 Accepted:

json

{
  "id": "crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f",
  "status": "queued",
  "urls": 3,
  "invalid_urls": []
}

Field	Type	Description
`id`	string	The job id. Use it with the crawl endpoints.
`status`	string	`queued`.
`urls`	integer	Number of URLs that will be scraped, after removing blanks, duplicates and skipped invalid entries.
`invalid_urls`	array of strings	Entries skipped because they are not valid URLs.

Results

GET /orgs/{id}/crawls/{crawlId} returns the job in the crawl format, with url set to batch: followed by the first URL and maxDepth set to 0. Once the job is finished, results lists one page per URL in the order you submitted them; a URL that could not be fetched is a page with an error. The job's status is success when at least one URL succeeded.

To see only the failures, use GET /orgs/{id}/crawls/{crawlId}/errors. To stop the job, DELETE /orgs/{id}/crawls/{crawlId}; URLs not yet fetched are reported with the error cancelled before fetch.

Limits and usage

100 URLs per batch, fetched four at a time.
The job runs for at most 2 minutes; URLs not fetched by then are reported as failed pages.
Each page is read up to 5 MB.

A batch counts as one crawl when submitted, and its pages, including failed ones, count against your monthly page quota when it finishes. If the quota is already used up, the batch is refused with 402.

Errors

Status	Code	When
`400`		`urls` is empty or has more than 100 entries, no valid URL remains, a format is invalid or not allowed, or a header, `webhook_events` entry or `max_age` is invalid.
`400`		Some entries are invalid and `ignore_invalid_urls` is not set. The body lists them in `invalid_urls`.
`400`	`WEBHOOK_TARGET_DENIED`	`webhook_url` is not a public URL.
`400`	`SIGNED_WEBHOOK_REQUIRED`	`webhook_url` was set without `webhook_secret`.
`402`		The monthly page quota is used up.
`403`		The key has neither the `scrape` nor the `crawl` scope.
`429`		Your plan's request rate limit was exceeded. Retry after `Retry-After`.
`501`		An LLM format (`json`, `product`, `menu`) was requested and no LLM is configured.
`503`		The job could not be queued.