# Batch scrape

Scrape a list of up to 100 known URLs as one asynchronous job. Each URL is fetched on its own, without following links, through the same pipeline as a crawl. The job has a crawl id, so you follow it, cancel it and list its failures with the [crawl endpoints](https://xdive.dev/docs/crawl.md#get-a-crawl), and it can notify a webhook.

`POST https://xdive.dev/api/orgs/{id}/batch/scrape`

Requires an API key with the `scrape` or the `crawl` scope. Submitting a batch counts against your plan's request rate limit.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/batch/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": [
      "https://example.com/pricing",
      "https://example.com/about",
      "https://example.com/blog/launch"
    ],
    "formats": ["markdown", "links"],
    "ignore_invalid_urls": true
  }'
# {"id":"crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f","status":"queued","urls":3,"invalid_urls":[]}
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
record = xdive.batch_scrape(
    [
        "https://example.com/pricing",
        "https://example.com/about",
        "https://example.com/blog/launch",
    ],
    formats=["markdown", "links"],
    ignore_invalid_urls=True,
)
for page in record["results"]:
    print(page["url"], page.get("error", ""))

# wait=False returns the job id instead of waiting:
job_id = xdive.batch_scrape(["https://example.com/pricing"], wait=False)
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const record = await xdive.batchScrape(
  [
    "https://example.com/pricing",
    "https://example.com/about",
    "https://example.com/blog/launch",
  ],
  { formats: ["markdown", "links"], ignore_invalid_urls: true },
);
for (const page of record.results ?? []) console.log(page.url, page.error ?? "");
```

The SDK methods submit the batch and poll it until it finishes (`wait=False` in Python returns the id instead). With cURL, poll `GET /orgs/{id}/crawls/{crawlId}` as described on [Crawl](https://xdive.dev/docs/crawl.md#polling-with-etag-and-retry-after).

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | required | 1 to 100 absolute `http` or `https` URLs. Blank entries are skipped and exact duplicates are scraped once. |
| `ignore_invalid_urls` | boolean | `false` | Skip invalid entries instead of rejecting the request. Skipped entries are returned in `invalid_urls`. |
| `formats` | array | none | Representations for each page, as on [Scrape](https://xdive.dev/docs/scrape.md#formats). `screenshot` and `audio` are not available. Without `formats`, each page has a `markdown` field. |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip URLs that robots.txt disallows. |
| `adaptive_fetch` | boolean | `true` | Plain HTTP first, escalating to a Chrome TLS fingerprint and then a headless browser for blocked pages, challenges and JavaScript shells. `false` uses plain HTTP only. |
| `bypass_cache` | boolean | `false` | Neither read nor write the page cache. |
| `max_age` | integer | `0` | Cache freshness window in milliseconds, as on [Scrape](https://xdive.dev/docs/scrape.md#caching-and-max_age). |
| `headers` | object | none | Extra request headers for every fetch, with the rules on [Scrape](https://xdive.dev/docs/scrape.md#custom-headers). |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses with typed tokens. |
| `webhook_url` | string | none | Receives signed event POSTs, as for [crawls](https://xdive.dev/docs/crawl.md#webhooks). |
| `webhook_secret` | string | none | HMAC-SHA256 signing key. Required unless the platform's signed-webhook policy is off. |
| `webhook_events` | array of strings | `["completed", "failed"]` | Any of `started`, `page`, `completed`, `failed`. |

A URL is valid when it parses as an absolute URL with an `http` or `https` scheme and a host.

## Response

The batch is queued and the response is `202 Accepted`:

```json
{
  "id": "crl_2b8e5d1f9a4c7e3b6d0f2a8c4e6b1d9f",
  "status": "queued",
  "urls": 3,
  "invalid_urls": []
}
```

| Field | Type | Description |
|---|---|---|
| `id` | string | The job id. Use it with the crawl endpoints. |
| `status` | string | `queued`. |
| `urls` | integer | Number of URLs that will be scraped, after removing blanks, duplicates and skipped invalid entries. |
| `invalid_urls` | array of strings | Entries skipped because they are not valid URLs. |

### Results

`GET /orgs/{id}/crawls/{crawlId}` returns the job in the [crawl format](https://xdive.dev/docs/crawl.md#get-a-crawl), with `url` set to `batch:` followed by the first URL and `maxDepth` set to `0`. Once the job is finished, `results` lists one page per URL in the order you submitted them; a URL that could not be fetched is a page with an `error`. The job's status is `success` when at least one URL succeeded.

To see only the failures, use [`GET /orgs/{id}/crawls/{crawlId}/errors`](https://xdive.dev/docs/crawl.md#list-failed-pages). To stop the job, [`DELETE /orgs/{id}/crawls/{crawlId}`](https://xdive.dev/docs/crawl.md#cancel-a-crawl); URLs not yet fetched are reported with the error `cancelled before fetch`.

## Limits and usage

- 100 URLs per batch, fetched four at a time.
- The job runs for at most 2 minutes; URLs not fetched by then are reported as failed pages.
- Each page is read up to 5 MB.

A batch counts as one crawl when submitted, and its pages, including failed ones, count against your monthly page quota when it finishes. If the quota is already used up, the batch is refused with `402`.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `urls` is empty or has more than 100 entries, no valid URL remains, a format is invalid or not allowed, or a header, `webhook_events` entry or `max_age` is invalid. |
| `400` | | Some entries are invalid and `ignore_invalid_urls` is not set. The body lists them in `invalid_urls`. |
| `400` | `WEBHOOK_TARGET_DENIED` | `webhook_url` is not a public URL. |
| `400` | `SIGNED_WEBHOOK_REQUIRED` | `webhook_url` was set without `webhook_secret`. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `scrape` nor the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `501` | | An LLM format (`json`, `product`, `menu`) was requested and no LLM is configured. |
| `503` | | The job could not be queued. |

```json
{
  "error": "urls contains invalid entries; pass ignore_invalid_urls to skip them",
  "invalid_urls": ["ftp://example.com/file", "example.com/no-scheme"]
}
```

## Related

- [Scrape](https://xdive.dev/docs/scrape.md) for one page, synchronously, with every option.
- [Crawl](https://xdive.dev/docs/crawl.md) for status, cancellation, errors and webhooks.
- [Extract](https://xdive.dev/docs/extract.md) to pull structured data across up to 10 URLs.

---

Source: https://xdive.dev/docs/batch-scrape · All docs for agents: https://xdive.dev/llms.txt