# Extract

Pull structured data out of up to 10 pages in one call. The pages are scraped concurrently, their content is combined, and your LLM answers a prompt or fills a JSON Schema from all of them together. Use Extract when the data you need is spread across a few known pages, such as a product's pricing, docs and changelog; for one page, the [`json` format on Scrape](https://xdive.dev/docs/scrape.md#formats) does the same per page.

`POST https://xdive.dev/api/orgs/{id}/extract`

Requires an API key with the `scrape` or the `crawl` scope, and an LLM configured for your organization. Extract is synchronous and stops after 90 seconds.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/extract" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{
    "urls": ["https://example.com/pricing", "https://example.com/docs/limits"],
    "prompt": "List each plan with its monthly price and API rate limit.",
    "schema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "monthly_price_usd": { "type": "number" },
              "requests_per_minute": { "type": "integer" }
            }
          }
        }
      }
    },
    "show_sources": true
  }'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
result = xdive.extract(
    ["https://example.com/pricing", "https://example.com/docs/limits"],
    prompt="List each plan with its monthly price and API rate limit.",
    schema={
        "type": "object",
        "properties": {
            "plans": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": {"type": "string"},
                        "monthly_price_usd": {"type": "number"},
                        "requests_per_minute": {"type": "integer"},
                    },
                },
            }
        },
    },
    show_sources=True,
)
print(result["data"]["plans"])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const result = await xdive.extract(["https://example.com/pricing", "https://example.com/docs/limits"], {
  prompt: "List each plan with its monthly price and API rate limit.",
  schema: {
    type: "object",
    properties: {
      plans: {
        type: "array",
        items: {
          type: "object",
          properties: {
            name: { type: "string" },
            monthly_price_usd: { type: "number" },
            requests_per_minute: { type: "integer" },
          },
        },
      },
    },
  },
  show_sources: true,
});
console.log(result.data);
```

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | array of strings | required | 1 to 10 absolute `http` or `https` URLs. |
| `prompt` | string | none | What to extract. At least one of `prompt` and `schema` is required. |
| `schema` | object | none | A JSON Schema for the result. Without a prompt, the model is asked to extract the data the schema describes. |
| `main_content_only` | boolean | `true` | Main-content extraction on each page: drop navigation, headers and footers. |
| `respect_robots` | boolean | `true` | Skip pages that robots.txt disallows. Such pages count as failed sources. |
| `show_sources` | boolean | `false` | Include the per-URL `sources` list in a successful response. |

## How it works

1. Each URL is scraped as Markdown through the standard pipeline (page cache, robots.txt, the same fetch escalation as [Scrape](https://xdive.dev/docs/scrape.md#fetch-telemetry)), three at a time, with 25 seconds per page.
2. Pages that fail are skipped. The others are combined in the order of `urls`, each labelled with its number and URL and cut to its first 24,000 bytes, so one long page cannot crowd out the rest.
3. The model fills `schema` or answers `prompt` from the combined text and returns JSON.

The request succeeds as long as one page was scraped. Use `show_sources` to see which pages failed and why.

## Response

```json
{
  "success": true,
  "data": {
    "plans": [
      { "name": "Starter", "monthly_price_usd": 0, "requests_per_minute": 60 },
      { "name": "Team", "monthly_price_usd": 49, "requests_per_minute": 600 }
    ]
  },
  "sources_scraped": 2,
  "sources": [
    { "url": "https://example.com/pricing", "title": "Pricing" },
    { "url": "https://example.com/docs/limits", "title": "Rate limits" }
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `success` | boolean | `true`. |
| `data` | object | The JSON the model produced, following `schema` when you gave one. |
| `sources_scraped` | integer | Pages that were scraped and used. |
| `sources` | array | With `show_sources`: one entry per URL with `url`, and `title` on success or `error` on failure. |

## Usage

Each page scraped counts as one page against your monthly page quota; pages that failed do not. The request is refused with `402` when the quota is already used up. The model call runs on your organization's LLM.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `urls` is empty, has more than 10 entries or contains an invalid URL, or neither `prompt` nor `schema` was given. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key has neither the `scrape` nor the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `501` | | No LLM is configured for the organization. |
| `502` | | No page could be scraped, or the model call failed (`extraction failed: …`). |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

When no page could be scraped, the body lists every URL and its error:

```json
{
  "success": false,
  "error": "no source could be scraped",
  "sources": [
    { "url": "https://example.com/pricing", "error": "disallowed by robots.txt" },
    { "url": "https://example.com/docs/limits", "error": "unexpected status 404" }
  ]
}
```

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and retries.

## Related

- [Scrape](https://xdive.dev/docs/scrape.md) with the `json`, `product` or `menu` format for one page.
- [Crawl](https://xdive.dev/docs/crawl.md) with `llm_extract` for records from every page of a site.
- [Research](https://xdive.dev/docs/research.md) with `schema` to fill a schema from pages found by searching.

---

Source: https://xdive.dev/docs/extract · All docs for agents: https://xdive.dev/llms.txt