# Errors and limits

How the API reports failures, what each status code and error code means, and the rate limits, concurrency limits and monthly quotas that apply to every organization. Read this before putting an integration into production.

## Error format

Every error is JSON with the same fields:

```json
{
  "success": false,
  "code": "CONCURRENCY_LIMIT",
  "error": "too many concurrent requests for this organization (plan limit 5); retry shortly",
  "retryable": true,
  "docs_url": "https://xdive.dev/docs/errors#error-codes"
}
```

| Field | Meaning |
|---|---|
| `code` | A stable identifier. Specific failures have their own code (see [Error codes](#error-codes)); every other error gets the code for its status, such as `INVALID_REQUEST`, `UNAUTHORIZED`, `QUOTA_EXCEEDED` or `NOT_FOUND`. |
| `error` | A message written for people. It may change; do not parse it. |
| `retryable` | `true` when sending the same request again later can succeed: rate limits, overload, timeouts and failures of a site or service the request depends on. `false` means the request, the credentials or the account must change first. |
| `docs_url` | This page. |

Branch on the HTTP status and on `code`, and retry only when `retryable` is `true`, after `Retry-After` seconds when the header is present.

Every response carries an `X-Request-ID` header (`req_…`). Include it when you report a problem.

A few endpoints add detail to the error body:

- `POST /orgs/{id}/batch/scrape` with malformed URLs returns `400` with an `invalid_urls` array, unless you pass `ignore_invalid_urls`.
- `POST /orgs/{id}/extract` returns `502` with a `sources` array giving each URL's error when no source could be read.
- The OAuth endpoints use the OAuth shape, `{"error": "invalid_grant", "error_description": "…"}`, and the MCP endpoint returns JSON-RPC errors.

Crawls and batch scrapes can succeed while some of their pages fail. Each failed page in the results has an `error` message and, for failures you may want to branch on, an `error_code` such as `SITE_RESTRICTED` or `PROXY_NOT_ENABLED`.

## HTTP status codes

| Status | Meaning in this API |
|---|---|
| `200` | Success. |
| `201` | Created (an API key, a browser session). |
| `202` | Accepted: a crawl or batch scrape was queued. Poll its id for the result. |
| `204` | Success with no body (revocations and deletions). |
| `304` | The crawl has not changed since the `ETag` you sent in `If-None-Match`. |
| `400` | The request is malformed or a parameter is invalid. The message says which. |
| `401` | No credentials, or the credentials are invalid, expired, used from an address outside the key's allowlist, or belong to another organization. |
| `402` | The organization's monthly quota for this kind of work is used up. See [Quotas](#quotas). |
| `403` | Authenticated but not allowed: the key lacks a scope, the user is not a member of the organization, the organization is suspended, the account is pending or rejected, or the target page is refused (robots.txt, `SITE_RESTRICTED`, `PROXY_NOT_ENABLED`). |
| `404` | The resource does not exist, or the work found nothing (`NO_SOURCES`, `SCRAPE_LOCKDOWN_CACHE_MISS`). |
| `405` | `GET` or `DELETE` on the MCP endpoint, which accepts `POST` only. |
| `409` | Conflict with the current state: cancelling a finished crawl, exporting an unfinished one, no free browser session. |
| `410` | The browser session has expired. |
| `415` | `POST /orgs/{id}/parse` was not sent as `multipart/form-data`. |
| `422` | The content could not be processed, for example a document that cannot be parsed. |
| `429` | Too many requests. Wait for `Retry-After` seconds. See [Rate limits and concurrency](#rate-limits-and-concurrency). |
| `500` | An unexpected server error. |
| `501` | The feature is not configured on this server: a language model, browser automation or billing. |
| `502` | Something the request depends on failed: the target site, a search engine or the language model. |
| `503` | The server is at capacity (`SERVER_BUSY`) or a dependency is unavailable, such as the job queue or an unconfigured search engine. |
| `504` | The work did not finish in time (`RESEARCH_TIMEOUT`). |

## Error codes

| Code | Status | Returned by | Meaning |
|---|---|---|---|
| `CONCURRENCY_LIMIT` | 429 | Synchronous endpoints | The organization already has its plan's maximum number of synchronous requests running. Sent with `Retry-After: 1`. |
| `SERVER_BUSY` | 503 | Synchronous endpoints | This API server is at capacity and its short wait queue is full. Sent with `Retry-After: 2`. |
| `PROXY_NOT_ENABLED` | 403 | Scrape; `error_code` on crawl pages | The site blocks data-centre networks, and the residential proxy that can reach it is not enabled for your organization, or your organization has used its monthly proxy allowance. The platform administrator enables the proxy and sets the allowance per organization. |
| `SITE_RESTRICTED` | 403 | Scrape; `error_code` on crawl pages | Your organization may use the residential proxy, but the proxy provider refused this destination. Providers typically refuse government and banking sites until the platform's proxy account is identity-verified. Retrying will not help; the platform administrator can see more in the audit log. |
| `SCRAPE_LOCKDOWN_CACHE_MISS` | 404 | Scrape with `lockdown` | Lockdown allows cache reads only, and no cached copy of the page exists. |
| `NO_SOURCES` | 404 | Research | No readable sources were found for the question. |
| `RESEARCH_TIMEOUT` | 504 | Research | The run did not finish in time (45 seconds for quick, 150 for deep). Try `depth: "quick"` or a narrower question. |
| `ACCOUNT_PENDING_APPROVAL` | 403 | Sign-in, sessions, OAuth, MCP | The account is waiting for an administrator's approval. |
| `ACCOUNT_REJECTED` | 403 | Sign-in, sessions, OAuth, MCP | The account's access request was not approved. |
| `EMAIL_NOT_VERIFIED` | 403 | Sign-in | An administrator address listed in `SUPER_ADMIN_EMAILS` has not been verified yet; a new verification link is emailed. |
| `SIGNED_WEBHOOK_REQUIRED` | 400 | Crawl, batch scrape, schedules | A `webhook_url` was given without a `webhook_secret` while the server requires signed webhooks (the default). |
| `WEBHOOK_TARGET_DENIED` | 400 | Crawl, batch scrape, schedules | The webhook URL is not an absolute public `http(s)` URL, for example it points at a private address. |
| `CORS_ORIGIN_DENIED` | 403 | Any endpoint called from a browser | The page's origin is not on the server's allowlist. |
| `RATE_LIMITED` | 429 | Any endpoint | The per-client or per-organization request rate was exceeded. Wait for `Retry-After` seconds. |
| `QUOTA_EXCEEDED` | 402 | Work-creating endpoints | The organization's monthly quota, or the API key's own monthly budget, is used up. The message says which. |
| `INVALID_IDEMPOTENCY_KEY` | 400 | Work-creating `POST` endpoints | The `Idempotency-Key` header is longer than 255 characters or has characters outside printable ASCII. |
| `IDEMPOTENCY_KEY_REUSED` | 422 | Work-creating `POST` endpoints | The key was already used with a different request (another path or body). Use a new key for a new request. |
| `IDEMPOTENCY_KEY_IN_PROGRESS` | 409 | Work-creating `POST` endpoints | The first request with this key is still running. Retry after `Retry-After` seconds to receive its response. |

## Rate limits and concurrency

Four limits protect the service. Each answers with `429` or `503` and a `Retry-After` header in seconds.

| Limit | Applies to | Over the limit |
|---|---|---|
| Per client | Every request: by default 20 per second per API key, or per IP address for requests without a key. | `429` `RATE_LIMITED`, `Retry-After: 1` |
| Requests per minute | Work-creating requests, counted per organization. | `429` `RATE_LIMITED`, with `Retry-After` until the minute resets |
| Concurrent requests | Synchronous requests running at once, per organization. | `429` `CONCURRENCY_LIMIT`, `Retry-After: 1` |
| Server capacity | Synchronous requests running on one API server. | `503` `SERVER_BUSY`, `Retry-After: 2` |

**Requests per minute** counts the calls that create work: scrape, crawl, batch scrape, map, search, research, extract, llms.txt, parse, connectors, and creating or driving browser sessions. Status polling, cancellation and other reads are never counted, so polling a running crawl cannot use up the budget. The window is a fixed minute. These responses carry the budget in the standard `RateLimit-Limit`, `RateLimit-Remaining` and `RateLimit-Reset` headers (seconds until the window resets), and under the same names with an `X-` prefix for older clients. Pace requests from `RateLimit-Remaining` rather than waiting for a `429`.

**Concurrent requests** and **server capacity** apply to the endpoints that fetch while you wait: scrape, map, search, research, extract, llms.txt, parse and connectors. Crawls and batch scrapes are queued instead and never rejected for concurrency: while other organizations' jobs are waiting, an organization gets its plan's fair share of crawl workers, and with nothing else queued it can use more.

Each API server runs up to 128 synchronous requests at once and lets 1,024 more wait up to 5 seconds before answering `SERVER_BUSY`. These are defaults that self-hosted servers can change.

| Plan | Requests per minute | Concurrent synchronous requests | Concurrent crawls (fair share) |
|---|---|---|---|
| `free` | 60 | 5 | 2 |
| `pro` | 600 | 50 | 10 |
| `enterprise` | 6,000 | 500 | 50 |

## Retrying safely with Idempotency-Key

A timeout or a dropped connection does not tell you whether the request ran. Retrying a crawl or a research call blindly can start it twice and use the quota twice. Send an `Idempotency-Key` header on work-creating `POST` requests to make retries safe:

```bash
curl -X POST "$XDIVE_URL/api/orgs/$ORG_ID/crawl" \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Idempotency-Key: 6f1c2d4e-8a7b-4c3d-9e0f-1a2b3c4d5e6f" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_depth": 1}'
```

- Use a new random value, such as a UUID, for each new request, and the same value for every retry of it.
- For 24 hours, a retry with the same key and the same body returns the first response, with the header `Idempotent-Replayed: true`, without running the request or counting it again.
- A retry while the first request is still running gets `409` `IDEMPOTENCY_KEY_IN_PROGRESS` with `Retry-After`.
- The same key with a different path or body gets `422` `IDEMPOTENCY_KEY_REUSED`.
- Rate-limit (`429`) and server (`5xx`) responses are not stored, so a retry after them runs the request. Responses larger than 4 MB are delivered but not stored.
- Keys belong to the organization. Two organizations can use the same value without effect on each other.

## Quotas

Each plan has monthly quotas, counted per calendar month in UTC (see [Usage and billing](https://xdive.dev/docs/usage.md) for what counts):

| Plan | Pages per month | Searches per month | Browser seconds per month | Browser sessions at once | Schedules |
|---|---|---|---|---|---|
| `free` | 1,000 | 100 | 600 | 1 | 3 |
| `pro` | 50,000 | 10,000 | 36,000 | 3 | 50 |
| `enterprise` | Unlimited | Unlimited | Unlimited | 10 | Unlimited |

When a quota is used up, requests that would use more of it return `402`:

```json
{ "error": "monthly page quota for the free plan reached (1000 pages); upgrade to continue" }
```

Quotas are checked before work starts, so a crawl that starts under the limit runs to completion even if it ends above it. Creating a schedule over the plan's cap also returns `402`; starting a browser session when the plan's sessions are all in use returns `409`.

## Retries

- **`429` and `503`:** wait for `Retry-After` seconds, then retry.
- **`402`:** retrying will not help until the next month or a plan upgrade.
- **Other `4xx`:** fix the request; the same request fails again.
- **`502` and `504`:** a dependency failed or was slow. A retry may succeed, but a successful retry is metered like any other request.

### How the SDKs retry

Both SDKs retry `429` and `503` automatically. Each retry waits the response's `Retry-After` (1 second when it is missing), up to `max_retries` (Python) or `maxRetries` (JavaScript) times, default `3`. After that they raise `RateLimited` / `RateLimitedError`, which carries the wait the server asked for and the error `code`.

| Situation | Python | JavaScript |
|---|---|---|
| Still throttled after retries (`429`, `503`) | `RateLimited` (`retry_after`) | `RateLimitedError` (`retryAfterSeconds`) |
| Quota used up (`402`) | `QuotaExceeded` | `QuotaExceededError` |
| Any other error status | `XDiveError` | `XDiveError` |
| Crawl not finished within the wait timeout | `CrawlTimeout` | `CrawlTimeoutError` |

All of them extend `XDiveError`, which exposes `status`, `code` and the message.

**Python**

```python
from xdive import XDive, XDiveError, QuotaExceeded, RateLimited

xdive = XDive(max_retries=5)
try:
    page = xdive.scrape("https://example.com")
except RateLimited as error:
    print(f"still throttled ({error.code}); retry in {error.retry_after}s")
except QuotaExceeded as error:
    print("quota used up:", error.message)
except XDiveError as error:
    print(error.status, error.code, error.message)
```

**JavaScript**

```ts
import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive";

const xdive = new XDive({ maxRetries: 5 });
try {
  const page = await xdive.scrape("https://example.com");
} catch (error) {
  if (error instanceof RateLimitedError) {
    console.log(`still throttled (${error.code}); retry in ${error.retryAfterSeconds}s`);
  } else if (error instanceof QuotaExceededError) {
    console.log("quota used up:", error.message);
  } else if (error instanceof XDiveError) {
    console.log(error.status, error.code, error.message);
  } else {
    throw error;
  }
}
```

The SDKs retry every `429` and `503`, including a `503` from a server whose search engine is not configured, so such a call fails only after the retries.

---

Source: https://xdive.dev/docs/errors · All docs for agents: https://xdive.dev/llms.txt