Extract

Pull structured data out of up to 10 pages in one call. The pages are scraped concurrently, their content is combined, and your LLM answers a prompt or fills a JSON Schema from all of them together. Use Extract when the data you need is spread across a few known pages, such as a product's pricing, docs and changelog; for one page, the json format on Scrape does the same per page.

POST/orgs/{id}/extract

Requires an API key with the scrape or the crawl scope, and an LLM configured for your organization. Extract is synchronous and stops after 90 seconds.

bash

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/extract" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://example.com/docs/limits"],
    "prompt": "List each plan with its monthly price and API rate limit.",
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthly_price_usd": { "type": "number" },
              "requests_per_minute": { "type": "integer" }
            }
          }
        }
      }
    },
    "show_sources": true
  }'

python

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.extract(
    ["https://example.com/pricing", "https://example.com/docs/limits"],
    prompt="List each plan with its monthly price and API rate limit.",
    schema={
        "type": "object",
        "properties": {
            "plans": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "monthly_price_usd": {"type": "number"},
                        "requests_per_minute": {"type": "integer"},
                    },
                },
            }
        },
    },
    show_sources=True,
)
print(result["data"]["plans"])

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.extract(["https://example.com/pricing", "https://example.com/docs/limits"], {
  prompt: "List each plan with its monthly price and API rate limit.",
  schema: {
    type: "object",
    properties: {
      plans: {
        type: "array",
        items: {
          type: "object",
          properties: {
            name: { type: "string" },
            monthly_price_usd: { type: "number" },
            requests_per_minute: { type: "integer" },
          },
        },
      },
    },
  },
  show_sources: true,
});
console.log(result.data);

Parameters

Field	Type	Default	Description
`urls`	array of strings	required	1 to 10 absolute `http` or `https` URLs.
`prompt`	string	none	What to extract. At least one of `prompt` and `schema` is required.
`schema`	object	none	A JSON Schema for the result. Without a prompt, the model is asked to extract the data the schema describes.
`main_content_only`	boolean	`true`	Main-content extraction on each page: drop navigation, headers and footers.
`respect_robots`	boolean	`true`	Skip pages that robots.txt disallows. Such pages count as failed sources.
`show_sources`	boolean	`false`	Include the per-URL `sources` list in a successful response.

How it works

Each URL is scraped as Markdown through the standard pipeline (page cache, robots.txt, the same fetch escalation as Scrape), three at a time, with 25 seconds per page.
Pages that fail are skipped. The others are combined in the order of urls, each labelled with its number and URL and cut to its first 24,000 bytes, so one long page cannot crowd out the rest.
The model fills schema or answers prompt from the combined text and returns JSON.

The request succeeds as long as one page was scraped. Use show_sources to see which pages failed and why.

Response

json

{
  "success": true,
  "data": {
    "plans": [
      { "name": "Starter", "monthly_price_usd": 0, "requests_per_minute": 60 },
      { "name": "Team", "monthly_price_usd": 49, "requests_per_minute": 600 }
    ]
  },
  "sources_scraped": 2,
  "sources": [
    { "url": "https://example.com/pricing", "title": "Pricing" },
    { "url": "https://example.com/docs/limits", "title": "Rate limits" }
  ]
}

Field	Type	Description
`success`	boolean	`true`.
`data`	object	The JSON the model produced, following `schema` when you gave one.
`sources_scraped`	integer	Pages that were scraped and used.
`sources`	array	With `show_sources`: one entry per URL with `url`, and `title` on success or `error` on failure.

Usage

Each page scraped counts as one page against your monthly page quota; pages that failed do not. The request is refused with 402 when the quota is already used up. The model call runs on your organization's LLM.

Errors

Status	Code	When
`400`		`urls` is empty, has more than 10 entries or contains an invalid URL, or neither `prompt` nor `schema` was given.
`402`		The monthly page quota is used up.
`403`		The key has neither the `scrape` nor the `crawl` scope.
`429`		Your plan's request rate limit was exceeded. Retry after `Retry-After`.
`429`	`CONCURRENCY_LIMIT`	Too many concurrent requests for your plan. Retry after `Retry-After`.
`501`		No LLM is configured for the organization.
`502`		No page could be scraped, or the model call failed (`extraction failed: …`).
`503`	`SERVER_BUSY`	The server is at capacity. Retry after `Retry-After`.

When no page could be scraped, the body lists every URL and its error: