Errors and limits

How the API reports failures, what each status code and error code means, and the rate limits, concurrency limits and monthly quotas that apply to every organization. Read this before putting an integration into production.

Error format

Every error is JSON with the same fields:

{
  "success": false,
  "code": "CONCURRENCY_LIMIT",
  "error": "too many concurrent requests for this organization (plan limit 5); retry shortly",
  "retryable": true,
  "docs_url": "https://xdive.dev/docs/errors#error-codes"
}

Branch on the HTTP status and on code, and retry only when retryable is true, after Retry-After seconds when the header is present.

Every response carries an X-Request-ID header (req_…). Include it when you report a problem.

A few endpoints add detail to the error body:

• POST /orgs/{id}/batch/scrape with malformed URLs returns 400 with an invalid_urls array, unless you pass ignore_invalid_urls.
• POST /orgs/{id}/extract returns 502 with a sources array giving each URL's error when no source could be read.
• The OAuth endpoints use the OAuth shape, {"error": "invalid_grant", "error_description": "…"}, and the MCP endpoint returns JSON-RPC errors.

Crawls and batch scrapes can succeed while some of their pages fail. Each failed page in the results has an error message and, for failures you may want to branch on, an error_code such as SITE_RESTRICTED or PROXY_NOT_ENABLED.

HTTP status codes

Error codes

Rate limits and concurrency

Four limits protect the service. Each answers with 429 or 503 and a Retry-After header in seconds.

Requests per minute counts the calls that create work: scrape, crawl, batch scrape, map, search, research, extract, llms.txt, parse, connectors, and creating or driving browser sessions. Status polling, cancellation and other reads are never counted, so polling a running crawl cannot use up the budget. The window is a fixed minute. These responses carry the budget in the standard RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset headers (seconds until the window resets), and under the same names with an X- prefix for older clients. Pace requests from RateLimit-Remaining rather than waiting for a 429.

Concurrent requests and server capacity apply to the endpoints that fetch while you wait: scrape, map, search, research, extract, llms.txt, parse and connectors. Crawls and batch scrapes are queued instead and never rejected for concurrency: while other organizations' jobs are waiting, an organization gets its plan's fair share of crawl workers, and with nothing else queued it can use more.

Each API server runs up to 128 synchronous requests at once and lets 1,024 more wait up to 5 seconds before answering SERVER_BUSY. These are defaults that self-hosted servers can change.

Retrying safely with Idempotency-Key

A timeout or a dropped connection does not tell you whether the request ran. Retrying a crawl or a research call blindly can start it twice and use the quota twice. Send an Idempotency-Key header on work-creating POST requests to make retries safe:

curl -X POST "$XDIVE_URL/api/orgs/$ORG_ID/crawl" \
  -H "Authorization: Bearer $XDIVE_API_KEY" \
  -H "Idempotency-Key: 6f1c2d4e-8a7b-4c3d-9e0f-1a2b3c4d5e6f" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "max_depth": 1}'

• Use a new random value, such as a UUID, for each new request, and the same value for every retry of it.
• For 24 hours, a retry with the same key and the same body returns the first response, with the header Idempotent-Replayed: true, without running the request or counting it again.
• A retry while the first request is still running gets 409 IDEMPOTENCY_KEY_IN_PROGRESS with Retry-After.
• The same key with a different path or body gets 422 IDEMPOTENCY_KEY_REUSED.
• Rate-limit (429) and server (5xx) responses are not stored, so a retry after them runs the request. Responses larger than 4 MB are delivered but not stored.
• Keys belong to the organization. Two organizations can use the same value without effect on each other.

Quotas

Each plan has monthly quotas, counted per calendar month in UTC (see Usage and billing for what counts):

When a quota is used up, requests that would use more of it return 402:

{ "error": "monthly page quota for the free plan reached (1000 pages); upgrade to continue" }

Quotas are checked before work starts, so a crawl that starts under the limit runs to completion even if it ends above it. Creating a schedule over the plan's cap also returns 402; starting a browser session when the plan's sessions are all in use returns 409.

Retries

• 429 and 503: wait for Retry-After seconds, then retry.
• 402: retrying will not help until the next month or a plan upgrade.
• Other 4xx: fix the request; the same request fails again.
• 502 and 504: a dependency failed or was slow. A retry may succeed, but a successful retry is metered like any other request.

How the SDKs retry

Both SDKs retry 429 and 503 automatically. Each retry waits the response's Retry-After (1 second when it is missing), up to max_retries (Python) or maxRetries (JavaScript) times, default 3. After that they raise RateLimited / RateLimitedError, which carries the wait the server asked for and the error code.

All of them extend XDiveError, which exposes status, code and the message.

from xdive import XDive, XDiveError, QuotaExceeded, RateLimited
 
xdive = XDive(max_retries=5)
try:
    page = xdive.scrape("https://example.com")
except RateLimited as error:
    print(f"still throttled ({error.code}); retry in {error.retry_after}s")
except QuotaExceeded as error:
    print("quota used up:", error.message)
except XDiveError as error:
    print(error.status, error.code, error.message)

import { XDive, XDiveError, QuotaExceededError, RateLimitedError } from "xdive";
 
const xdive = new XDive({ maxRetries: 5 });
try {
  const page = await xdive.scrape("https://example.com");
} catch (error) {
  if (error instanceof RateLimitedError) {
    console.log(`still throttled (${error.code}); retry in ${error.retryAfterSeconds}s`);
  } else if (error instanceof QuotaExceededError) {
    console.log("quota used up:", error.message);
  } else if (error instanceof XDiveError) {
    console.log(error.status, error.code, error.message);
  } else {
    throw error;
  }
}

The SDKs retry every 429 and 503, including a 503 from a server whose search engine is not configured, so such a call fails only after the retries.