# Scrape

Fetch one URL and get it back as clean Markdown, HTML, plain text, links, images, a screenshot or structured data. Scrape is synchronous: the response carries the page. Use it when you need a single page now; use [Batch scrape](https://xdive.dev/docs/batch-scrape.md) for lists of URLs and [Crawl](https://xdive.dev/docs/crawl.md) to follow links across a site.

`POST https://xdive.dev/api/orgs/{id}/scrape`

Requires an API key with the `scrape` scope. A scrape times out after 30 seconds, or 3 minutes when `audio` is requested; [browser-backed scrapes](#browser-actions) allow 45 seconds per step. HTML, PDF, DOCX, XLSX and plain-text responses up to 50 MB are converted.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "links"],
    "max_age": 3600000
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape(
    "https://example.com/pricing",
    formats=["markdown", "links"],
    max_age=3_600_000,
)
print(page["artifacts"]["markdown"]["content"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com/pricing", {
  formats: ["markdown", "links"],
  max_age: 3_600_000,
});
console.log(page.artifacts?.markdown?.content);
```

The SDKs return the `data` object of the response. They look up your organization id once through `GET /whoami`, so you only configure the key.

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | Absolute `http` or `https` URL to fetch. |
| `formats` | array | `["markdown"]` | Representations to return. Each entry is a format name or an object with per-format options. See [Formats](#formats). |
| `extract` | boolean | `true` | Main-content extraction: drop navigation, headers, footers and sidebars. `false` converts the whole page. |
| `respect_robots` | boolean | `true` | Refuse URLs that the site's robots.txt disallows for the crawler. |
| `max_age` | integer | `0` | Freshness window in milliseconds. A cached copy younger than this is served without a network fetch. `0` uses the server's default window. See [Caching](#caching-and-max_age). |
| `max_chars` | integer | `0` | Return at most this many characters of Markdown, cut at a paragraph break. `0` means no limit; otherwise at least 500. See [Long pages](#long-pages). |
| `offset` | integer | `0` | Continue a long page from the previous response's `next_offset`. |
| `store_in_cache` | boolean | `true` | `false` bypasses the page cache for this request: the page is fetched live and not stored. |
| `headers` | object | none | Extra request headers for the page fetch, such as `Accept-Language`, a mobile `User-Agent` or `Cookie`. See [Custom headers](#custom-headers). |
| `actions` | array | none | Browser steps to run before the content is captured (click, fill, press, scroll, wait). At most 15. See [Browser actions](#browser-actions). |
| `wait_for` | integer | `0` | Milliseconds to wait after the page loads and the actions run, before capture. 0 to 10000. Uses the browser. |
| `full_page_screenshot` | boolean | `false` | With the `screenshot` format, capture the full scrollable page instead of the viewport. |
| `redact_pii` | boolean | `false` | Replace emails, phone numbers, card numbers, SSNs, IBANs and IP addresses with typed tokens. See [Privacy controls](#privacy-controls). |
| `zero_data_retention` | boolean | `false` | Do not write the page to the cache and mark the response `no-store`. |
| `lockdown` | boolean | `false` | Serve only an already-cached copy and make no outbound page fetch. Implies `zero_data_retention`. |

## Formats

Pass format names as strings, or objects when a format takes options. Each format may appear once. Names are case-insensitive and a few aliases are accepted (`md`, `txt`, `raw_html`, `clean_html`, `summarize`, `answer`, `screenshots`, `mp3`); the response always uses the canonical name.

| Format | Returned in the artifact | Notes |
|---|---|---|
| `markdown` | `content` | Clean Markdown. The default. |
| `text` | `content` | Plain text. |
| `html` | `content` | Clean HTML rendered from the parsed page (main content when `extract` is on). |
| `rawHtml` | `content` | The HTML as fetched. |
| `links` | `links`: list of `url` and `text` | Every link on the page, whatever `extract` is set to. |
| `images` | `images`: list of `url`, `alt`, `title` | Every image on the page. |
| `branding` | `branding` | Observed identity signals: `name`, `description`, `logo`, `favicon`, `colors`, `fonts`, `css_variables`. Absent values stay absent. |
| `summary` | `content` | A short summary: 3 to 6 sentences written by your LLM when one is configured, otherwise 5 sentences extracted from the page. Accepts `prompt` for extra instructions. |
| `question` | `content` | Answers `question` (or `prompt`) from the page only. Uses your LLM when configured, otherwise the most relevant sentences. |
| `highlights` | `highlights`: list of `text` and `score` | Sentences most relevant to `query` (or `prompt`), best first. No LLM. `max_results` sets the count (default 5). |
| `json` | `json` | Structured data from `prompt` and/or a JSON Schema in `schema`. Requires an LLM. |
| `product` | `json` | Product fields (title, brand, price, currency, availability, sku, variants…) from a built-in schema. Requires an LLM. |
| `menu` | `json` | Restaurant menu sections and items from a built-in schema. Requires an LLM. |
| `screenshot` | `content` | A PNG as a `data:image/png;base64,…` URL, at most 8 MiB. Runs in the browser. See [Screenshots](#screenshots). |
| `audio` | `url` | The audio track of an HTTPS YouTube or Vimeo URL as MP3, up to 100 MB. See [Audio](#audio). |

The LLM is the model configured for your organization in AI settings (or the platform's). Without one, `json`, `product` and `menu` fail with `501`. Text sent to the model is capped at 120,000 characters.

### Format options

| Option | Type | Used by | Description |
|---|---|---|---|
| `type` | string | all | The format name. |
| `prompt` | string | `json`, `summary`, `question`, `highlights` | Instructions for the format. For `question` and `highlights` it stands in for `question` and `query`. |
| `schema` | object | `json` | A JSON Schema for the result. Must be valid JSON. |
| `question` | string | `question` | The question to answer. `question` or `prompt` is required. |
| `query` | string | `highlights` | What the highlights should be relevant to. Without it, the page's most representative sentences are returned. |
| `max_results` | integer | `highlights` | Number of highlights, 0 to 50. |

```json
{
  "url": "https://example.com/product/espresso-grinder",
  "formats": [
    "markdown",
    { "type": "json", "prompt": "Extract the product", "schema": {
      "type": "object",
      "properties": { "name": { "type": "string" }, "price": { "type": "number" } }
    } },
    { "type": "highlights", "query": "warranty", "max_results": 3 }
  ]
}
```

### Screenshots

Requesting `screenshot` routes the scrape through an isolated, short-lived browser session instead of the standard fetch. The other formats you ask for are rendered from the same final DOM. Add `full_page_screenshot: true` for the whole page.

### Audio

`audio` extracts the audio of an HTTPS YouTube (`youtube.com`, `youtu.be`) or Vimeo URL. The artifact's `url` is a signed link to the MP3 that expires after one hour. Audio needs object storage on the server, cannot be combined with `zero_data_retention` or `lockdown`, and is not available on crawls or batch scrapes. If extraction fails the request returns `422`.

## Main-content extraction

With `extract` on (the default), the page's main article is found and navigation, headers, footers and sidebars are dropped before conversion, so `markdown`, `text` and `html` hold the content a reader came for. Set `extract: false` to convert the whole document. Metadata (`title`, `description`, `lang`…), `links` and `images` always describe the whole page.

## Caching and max_age

Pages are cached across the platform. On a fresh hit the response is served without a network fetch and the page reports `fetch_route: "cache"` with `fetch_cost_units: 0`.

- `max_age` sets how old a cached copy may be, in milliseconds. An older copy is fetched again from the site, and the fresh response replaces it. A window longer than the server default accepts older copies for as long as the cache still holds them.
- `store_in_cache: false` skips the cache in both directions for this request.
- `zero_data_retention` and `lockdown` never write to the cache.
- Responses fetched with custom `headers` are cached separately from header-less fetches, so a page fetched with your cookies is never served to a request without them.

## Robots.txt

By default the crawler checks the site's robots.txt and refuses disallowed URLs with `403` and `disallowed by robots.txt`. Set `respect_robots: false` to skip the check when you have the right to fetch the page.

## Long pages

Every Markdown artifact reports `estimated_tokens`, about four characters per token, so an agent knows what a page costs to read before it reads it. To read a long page in parts, pass `max_chars`. The artifact then carries `total_chars` and, while more remains, `next_offset`. Send the same request again with `offset` set to `next_offset` to get the next part. A continuation reuses the copy fetched for the first part (up to 10 minutes old), so all parts come from the same fetch. Each request is metered as one page.

```json
{ "url": "https://go.dev/doc/effective_go", "max_chars": 4000 }
```

```json
"markdown": { "content": "# Effective Go …", "estimated_tokens": 915, "total_chars": 102176, "next_offset": 3661 }
```

## JavaScript pages and warnings

A plain scrape starts with a fast HTTP fetch. When the result is a JavaScript application shell or loses most of the page's text, the server renders the page again in a browser and returns that instead; `fetch_route` is then `browser` and `fetch_reason` says why. The browser time is metered as browser seconds, on top of the page.

When the content still looks incomplete, for example because the browser quota is used up or the request sends custom `headers` (a browser cannot send them), the page succeeds with a `warnings` array explaining it. Retry with `wait_for` to give a slow page more time to render, or use what you got.

## Custom headers

`headers` adds up to 10 request headers to the page fetch. Names are 1 to 128 characters, values at most 2048 and without line breaks. `Host`, `Content-Length`, `Transfer-Encoding`, `Connection`, `Upgrade`, `TE`, `Trailer`, `Proxy-Authorization` and any `X-Forwarded-*` header are rejected. A `User-Agent` you set replaces the crawler's. Headers apply to the HTTP fetch; browser-backed scrapes do not send them.

## Browser actions

`actions`, `wait_for` and the `screenshot` format run the scrape in an isolated, ephemeral browser session: it opens `url`, runs your actions in order, waits `wait_for` milliseconds, captures the DOM and renders the requested formats from it. Each action has 45 seconds to complete.

| Action | Fields | Description |
|---|---|---|
| `click` | `selector` | Click the first element matching a CSS selector. |
| `fill` | `selector`, `value` | Wait for the element to be visible, focus it and set its value. |
| `press` | `key` | Press one of `Enter`, `Tab`, `Escape`, `Backspace`, `Delete`, `ArrowUp`, `ArrowDown`, `ArrowLeft`, `ArrowRight`, `Space`. |
| `scroll` | `x`, `y` | Scroll the window by this many pixels (each between -100000 and 100000). |
| `wait` | `selector` or `milliseconds` | Wait until the selector is visible, or sleep 1 to 10000 milliseconds. |

Selectors are at most 1000 characters and values at most 10000. `goto`, `extract` and `screenshot` actions are not accepted on scrape: the scrape navigates to `url` itself, and you request content and screenshots through `formats`.

```json
{
  "url": "https://example.com/search",
  "actions": [
    { "type": "fill", "selector": "input[name=q]", "value": "espresso" },
    { "type": "press", "key": "Enter" },
    { "type": "wait", "selector": ".results" }
  ],
  "wait_for": 1000,
  "formats": ["markdown", "screenshot"]
}
```

Browser-backed scrapes are metered as browser seconds (rounded up, at least one) against your plan's browser quota instead of pages. They need browser automation on the server (`501` otherwise) and a free browser session (`409` when the server's sessions, or your plan's, are all in use). `respect_robots` applies to the starting URL (on by default; a disallowed URL returns `403`). `headers` cannot be sent by the browser and are rejected with `400`; `max_age` and `store_in_cache` do not apply, since a browser scrape always renders live. They cannot be combined with `lockdown`. They are not available through `GET`.

## Privacy controls

**`redact_pii`** deterministically replaces machine-recognizable identifiers in the returned title, description and text artifacts (including `json` and `highlights`) with typed tokens: `[REDACTED:email]`, `[REDACTED:phone]`, `[REDACTED:card]` (Luhn-checked), `[REDACTED:ssn]`, `[REDACTED:iban]` and `[REDACTED:ip]`. Names and street addresses are not detected. Link and image URLs are not changed.

**`zero_data_retention`** keeps the page out of the cache, sets `Cache-Control: no-store, private` and `Pragma: no-cache`, and the audit log entry for a standard scrape omits the URL. Formats that use an LLM still send the page text to your configured model.

**`lockdown`** serves the page only if it is already cached, whatever its age. No DNS lookup, robots.txt fetch, HTTP request or browser session is made. A cache miss returns `404` with `SCRAPE_LOCKDOWN_CACHE_MISS`.

> **Warning:**
>
> Lockdown makes no model calls either. Request only formats rendered from the page (`markdown`, `text`, `html`, `rawHtml`, `links`, `images`, `branding`, `highlights`); asking for `json`, `product` or `menu`, or for `summary` or `question` when a model is configured, returns `400` before any work is done.

A platform administrator can make zero data retention or lockdown the default for every request. A successful POST reports the policy that applied in `security`, and both POST and GET responses carry two headers:

| Header | Values |
|---|---|
| `X-Data-Retention` | `none` with zero data retention, otherwise `standard` |
| `X-Network-Policy` | `cache_only` under lockdown, otherwise `live` |

## Response

```json
{
  "success": true,
  "data": {
    "url": "https://example.com/pricing",
    "title": "Pricing",
    "description": "Plans for teams of every size.",
    "lang": "en",
    "depth": 0,
    "status_code": 200,
    "word_count": 412,
    "fetched_at": "2026-10-15T09:21:44.118302Z",
    "fetch_route": "impersonate",
    "fetch_attempts": 2,
    "fetch_cost_units": 3,
    "fetch_reason": "blocked or transient HTTP status 403",
    "artifacts": {
      "markdown": {
        "media_type": "text/markdown; charset=utf-8",
        "content": "# Pricing\n\nStart free, then pay as you grow...",
        "bytes": 2841,
        "quality": { "passed": true, "score": 100 }
      },
      "links": {
        "media_type": "application/json",
        "links": [{ "url": "https://example.com/docs", "text": "Docs" }],
        "bytes": 96,
        "quality": { "passed": true, "score": 100 }
      }
    },
    "links": ["https://example.com/docs", "https://example.com/signup"]
  },
  "security": {
    "zero_data_retention": false,
    "network_policy": "live",
    "cache_state": "eligible",
    "outbound_permitted": true
  }
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true` on success. |
| `data` | object | The page. |
| `security` | object | The data policy that applied to this request. |

### The page object

| Field | Type | Description |
|---|---|---|
| `url` | string | Final URL after redirects. |
| `title`, `description`, `lang`, `canonical`, `image`, `site_name` | string | Page metadata, when present. |
| `depth` | integer | Always `0` for a scrape. |
| `status_code` | integer | HTTP status of the fetch. |
| `word_count` | integer | Words in the page's Markdown. |
| `fetched_at` | string | When the content was fetched (RFC 3339). |
| `fetch_route`, `fetch_attempts`, `fetch_cost_units`, `fetch_reason` | | How the page was fetched. See [Fetch telemetry](#fetch-telemetry). |
| `artifacts` | object | One artifact per requested format, keyed by format name. Only requested formats are returned. |
| `links` | array of strings | URLs of the links found on the page. |

Each artifact has `media_type`, `bytes`, a `quality` report (`passed`, `score` from 0 to 100, optional `warnings`) and the field for its format: `content`, `json`, `links`, `images`, `highlights`, `branding` or `url`.

### Fetch telemetry

Each page reports how it was fetched. A plain HTTP request is tried first; on a block, challenge or bot-filter response the fetch escalates to a Chrome TLS fingerprint, and on servers with an egress proxy, sites that refuse the server's network are retried through the proxy.

| Field | Description |
|---|---|
| `fetch_route` | The route that produced the content: `cache`, `http`, `impersonate`, `browser` or `proxy`. |
| `fetch_attempts` | Network attempts made, across routes. `0` for a cache hit. |
| `fetch_cost_units` | Relative cost of the attempts: 1 per HTTP attempt, 2 per impersonated attempt, 20 per browser render, plus 5 for the egress proxy. `0` for a cache hit. It is telemetry, not your bill: a scrape is metered as one page. |
| `fetch_reason` | Why the fetch escalated or where it came from, for example `blocked or transient HTTP status 403`, `JavaScript application shell detected` or `fresh response cache hit`. Empty when the first HTTP attempt succeeded. |

### The security object

| Field | Values |
|---|---|
| `zero_data_retention` | Whether zero data retention applied. |
| `network_policy` | `live`, or `cache_only` under lockdown. |
| `cache_state` | `eligible` (the page may be cached), `write_disabled` (it was not stored), or `hit` (lockdown served the cache). |
| `outbound_permitted` | `false` under lockdown. |

## Government and banking sites

Many government and banking sites refuse every connection from data-centre networks, so no fetch strategy reaches them from the crawler's own address. XDive can route those requests through a residential proxy. The proxy is paid per page, so it is off for every organization until the platform administrator enables it for yours, with a monthly allowance of proxied pages (200 by default). Your current allowance and use are in the `proxy` object of [`GET /orgs/{id}/usage`](https://xdive.dev/docs/usage.md).

With access enabled, scrapes, crawls and batch scrapes go direct first and use the proxy only for sites that cannot be reached or stay blocked. Proxied pages report `fetch_route: "proxy"`, and each one counts once against the allowance. When the allowance is used up, fetches go direct only until the next calendar month.

Two errors are specific to these sites:

- **`403 PROXY_NOT_ENABLED`**: the site blocks data-centre networks, and your organization has no proxy access or has used its allowance. Ask the platform administrator to enable the proxy, or to raise the allowance.
- **`403 SITE_RESTRICTED`**: the proxy is enabled for your organization, but the proxy provider refused the destination. Providers typically refuse government and banking sites until the platform's proxy account is identity-verified. Retrying will not help.

In crawl and batch results, the same codes appear as the failed page's `error_code`.

## GET (Markdown only)

`GET https://xdive.dev/api/orgs/{id}/scrape`

The original endpoint returns the page's Markdown as `text/markdown` instead of JSON. It accepts the query parameters `url`, `respect_robots`, `zero_data_retention` and `lockdown`, and the same errors as POST. Formats, actions, headers and `max_age` need POST.

```bash
curl "https://xdive.dev/api/orgs/$ORG_ID/scrape?url=https://example.com" \
  -H "X-API-Key: $XDIVE_API_KEY"
```

## Errors

Errors have an `error` message; the ones below that carry a `code` also have `"success": false`.

| Status | Code | When |
|---|---|---|
| `400` | | Missing `url`, an unknown or repeated format, invalid format options, an invalid header, a negative `max_age`, more than 15 actions, `wait_for` out of range, or an option combination that is not allowed. |
| `402` | | The monthly page quota (or browser quota, for browser scrapes) is used up. |
| `403` | | The key lacks the `scrape` scope, or robots.txt disallows the URL. |
| `403` | `PROXY_NOT_ENABLED` | The site blocks data-centre networks and your organization has no residential proxy access, or has used its monthly proxy allowance. See [Government and banking sites](#government-and-banking-sites). |
| `403` | `SITE_RESTRICTED` | Your organization may use the residential proxy, but the proxy provider refused this destination, which is common for government and banking sites. Retrying will not help. |
| `404` | `SCRAPE_LOCKDOWN_CACHE_MISS` | Lockdown is on and the page is not cached. |
| `409` | | No browser session is free for a browser-backed scrape. |
| `422` | | Audio extraction failed or the URL is not a supported video. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `501` | | A requested format needs an LLM that is not configured, or the server has no browser automation. |
| `502` | | The page could not be fetched or converted (`scrape failed: …`), or an enrichment format failed. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors, rate limits and retries.

## Related

- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) to scrape up to 100 URLs as one job.
- [Crawl](https://xdive.dev/docs/crawl.md) to follow links across a site.
- [Extract](https://xdive.dev/docs/extract.md) for structured data across several pages.
- [Usage and billing](https://xdive.dev/docs/usage.md) for what is metered.

---

Source: https://xdive.dev/docs/scrape · All docs for agents: https://xdive.dev/llms.txt