Scrape

Fetch one URL and get it back as clean Markdown, HTML, plain text, links, images, a screenshot or structured data. Scrape is synchronous: the response carries the page. Use it when you need a single page now; use Batch scrape for lists of URLs and Crawl to follow links across a site.

Requires an API key with the scrape scope. A scrape times out after 30 seconds, or 3 minutes when audio is requested; browser-backed scrapes allow 45 seconds per step. HTML, PDF, DOCX, XLSX and plain-text responses up to 50 MB are converted.

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/scrape" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown", "links"],
    "max_age": 3600000
  }'

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
page = xdive.scrape(
    "https://example.com/pricing",
    formats=["markdown", "links"],
    max_age=3_600_000,
)
print(page["artifacts"]["markdown"]["content"])

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const page = await xdive.scrape("https://example.com/pricing", {
  formats: ["markdown", "links"],
  max_age: 3_600_000,
});
console.log(page.artifacts?.markdown?.content);

The SDKs return the data object of the response. They look up your organization id once through GET /whoami, so you only configure the key.

Parameters

Formats

Pass format names as strings, or objects when a format takes options. Each format may appear once. Names are case-insensitive and a few aliases are accepted (md, txt, raw_html, clean_html, summarize, answer, screenshots, mp3); the response always uses the canonical name.

The LLM is the model configured for your organization in AI settings (or the platform's). Without one, json, product and menu fail with 501. Text sent to the model is capped at 120,000 characters.

Format options

{
  "url": "https://example.com/product/espresso-grinder",
  "formats": [
    "markdown",
    { "type": "json", "prompt": "Extract the product", "schema": {
      "type": "object",
      "properties": { "name": { "type": "string" }, "price": { "type": "number" } }
    } },
    { "type": "highlights", "query": "warranty", "max_results": 3 }
  ]
}

Screenshots

Requesting screenshot routes the scrape through an isolated, short-lived browser session instead of the standard fetch. The other formats you ask for are rendered from the same final DOM. Add full_page_screenshot: true for the whole page.

Audio

audio extracts the audio of an HTTPS YouTube (youtube.com, youtu.be) or Vimeo URL. The artifact's url is a signed link to the MP3 that expires after one hour. Audio needs object storage on the server, cannot be combined with zero_data_retention or lockdown, and is not available on crawls or batch scrapes. If extraction fails the request returns 422.

Main-content extraction

With extract on (the default), the page's main article is found and navigation, headers, footers and sidebars are dropped before conversion, so markdown, text and html hold the content a reader came for. Set extract: false to convert the whole document. Metadata (title, description, lang…), links and images always describe the whole page.

Caching and max_age

Pages are cached across the platform. On a fresh hit the response is served without a network fetch and the page reports fetch_route: "cache" with fetch_cost_units: 0.

• max_age sets how old a cached copy may be, in milliseconds. An older copy is fetched again from the site, and the fresh response replaces it. A window longer than the server default accepts older copies for as long as the cache still holds them.
• store_in_cache: false skips the cache in both directions for this request.
• zero_data_retention and lockdown never write to the cache.
• Responses fetched with custom headers are cached separately from header-less fetches, so a page fetched with your cookies is never served to a request without them.

Robots.txt

By default the crawler checks the site's robots.txt and refuses disallowed URLs with 403 and disallowed by robots.txt. Set respect_robots: false to skip the check when you have the right to fetch the page.

Long pages

Every Markdown artifact reports estimated_tokens, about four characters per token, so an agent knows what a page costs to read before it reads it. To read a long page in parts, pass max_chars. The artifact then carries total_chars and, while more remains, next_offset. Send the same request again with offset set to next_offset to get the next part. A continuation reuses the copy fetched for the first part (up to 10 minutes old), so all parts come from the same fetch. Each request is metered as one page.

{ "url": "https://go.dev/doc/effective_go", "max_chars": 4000 }

"markdown": { "content": "# Effective Go …", "estimated_tokens": 915, "total_chars": 102176, "next_offset": 3661 }

JavaScript pages and warnings

A plain scrape starts with a fast HTTP fetch. When the result is a JavaScript application shell or loses most of the page's text, the server renders the page again in a browser and returns that instead; fetch_route is then browser and fetch_reason says why. The browser time is metered as browser seconds, on top of the page.

When the content still looks incomplete, for example because the browser quota is used up or the request sends custom headers (a browser cannot send them), the page succeeds with a warnings array explaining it. Retry with wait_for to give a slow page more time to render, or use what you got.

Custom headers

headers adds up to 10 request headers to the page fetch. Names are 1 to 128 characters, values at most 2048 and without line breaks. Host, Content-Length, Transfer-Encoding, Connection, Upgrade, TE, Trailer, Proxy-Authorization and any X-Forwarded-* header are rejected. A User-Agent you set replaces the crawler's. Headers apply to the HTTP fetch; browser-backed scrapes do not send them.

Browser actions

actions, wait_for and the screenshot format run the scrape in an isolated, ephemeral browser session: it opens url, runs your actions in order, waits wait_for milliseconds, captures the DOM and renders the requested formats from it. Each action has 45 seconds to complete.

Selectors are at most 1000 characters and values at most 10000. goto, extract and screenshot actions are not accepted on scrape: the scrape navigates to url itself, and you request content and screenshots through formats.

{
  "url": "https://example.com/search",
  "actions": [
    { "type": "fill", "selector": "input[name=q]", "value": "espresso" },
    { "type": "press", "key": "Enter" },
    { "type": "wait", "selector": ".results" }
  ],
  "wait_for": 1000,
  "formats": ["markdown", "screenshot"]
}

Browser-backed scrapes are metered as browser seconds (rounded up, at least one) against your plan's browser quota instead of pages. They need browser automation on the server (501 otherwise) and a free browser session (409 when the server's sessions, or your plan's, are all in use). respect_robots applies to the starting URL (on by default; a disallowed URL returns 403). headers cannot be sent by the browser and are rejected with 400; max_age and store_in_cache do not apply, since a browser scrape always renders live. They cannot be combined with lockdown. They are not available through GET.

Privacy controls

redact_pii deterministically replaces machine-recognizable identifiers in the returned title, description and text artifacts (including json and highlights) with typed tokens: [REDACTED:email], [REDACTED:phone], [REDACTED:card] (Luhn-checked), [REDACTED:ssn], [REDACTED:iban] and [REDACTED:ip]. Names and street addresses are not detected. Link and image URLs are not changed.

zero_data_retention keeps the page out of the cache, sets Cache-Control: no-store, private and Pragma: no-cache, and the audit log entry for a standard scrape omits the URL. Formats that use an LLM still send the page text to your configured model.

lockdown serves the page only if it is already cached, whatever its age. No DNS lookup, robots.txt fetch, HTTP request or browser session is made. A cache miss returns 404 with SCRAPE_LOCKDOWN_CACHE_MISS.

A platform administrator can make zero data retention or lockdown the default for every request. A successful POST reports the policy that applied in security, and both POST and GET responses carry two headers:

Response

{
  "success": true,
  "data": {
    "url": "https://example.com/pricing",
    "title": "Pricing",
    "description": "Plans for teams of every size.",
    "lang": "en",
    "depth": 0,
    "status_code": 200,
    "word_count": 412,
    "fetched_at": "2026-10-15T09:21:44.118302Z",
    "fetch_route": "impersonate",
    "fetch_attempts": 2,
    "fetch_cost_units": 3,
    "fetch_reason": "blocked or transient HTTP status 403",
    "artifacts": {
      "markdown": {
        "media_type": "text/markdown; charset=utf-8",
        "content": "# Pricing\n\nStart free, then pay as you grow...",
        "bytes": 2841,
        "quality": { "passed": true, "score": 100 }
      },
      "links": {
        "media_type": "application/json",
        "links": [{ "url": "https://example.com/docs", "text": "Docs" }],
        "bytes": 96,
        "quality": { "passed": true, "score": 100 }
      }
    },
    "links": ["https://example.com/docs", "https://example.com/signup"]
  },
  "security": {
    "zero_data_retention": false,
    "network_policy": "live",
    "cache_state": "eligible",
    "outbound_permitted": true
  }
}

The page object

Each artifact has media_type, bytes, a quality report (passed, score from 0 to 100, optional warnings) and the field for its format: content, json, links, images, highlights, branding or url.

Fetch telemetry

Each page reports how it was fetched. A plain HTTP request is tried first; on a block, challenge or bot-filter response the fetch escalates to a Chrome TLS fingerprint, and on servers with an egress proxy, sites that refuse the server's network are retried through the proxy.

The security object

Government and banking sites

Many government and banking sites refuse every connection from data-centre networks, so no fetch strategy reaches them from the crawler's own address. XDive can route those requests through a residential proxy. The proxy is paid per page, so it is off for every organization until the platform administrator enables it for yours, with a monthly allowance of proxied pages (200 by default). Your current allowance and use are in the proxy object of GET /orgs/{id}/usage.

With access enabled, scrapes, crawls and batch scrapes go direct first and use the proxy only for sites that cannot be reached or stay blocked. Proxied pages report fetch_route: "proxy", and each one counts once against the allowance. When the allowance is used up, fetches go direct only until the next calendar month.

Two errors are specific to these sites:

• 403 PROXY_NOT_ENABLED: the site blocks data-centre networks, and your organization has no proxy access or has used its allowance. Ask the platform administrator to enable the proxy, or to raise the allowance.
• 403 SITE_RESTRICTED: the proxy is enabled for your organization, but the proxy provider refused the destination. Providers typically refuse government and banking sites until the platform's proxy account is identity-verified. Retrying will not help.

In crawl and batch results, the same codes appear as the failed page's error_code.

GET (Markdown only)

The original endpoint returns the page's Markdown as text/markdown instead of JSON. It accepts the query parameters url, respect_robots, zero_data_retention and lockdown, and the same errors as POST. Formats, actions, headers and max_age need POST.

curl "https://xdive.dev/api/orgs/$ORG_ID/scrape?url=https://example.com" \
  -H "X-API-Key: $XDIVE_API_KEY"

Errors

Errors have an error message; the ones below that carry a code also have "success": false.

See Errors and limits for authentication errors, rate limits and retries.

Related

• Batch scrape to scrape up to 100 URLs as one job.
• Crawl to follow links across a site.
• Extract for structured data across several pages.
• Usage and billing for what is metered.