Self-hosting

The whole platform (API, crawl workers, dashboard and these docs) is in one repository and runs on your own servers with Docker Compose. This page covers the essentials; the repository guides have the full detail:

• RUNNING.md: step-by-step local setup, with Docker or from source.
• README.md: features, configuration and deployment notes.
• OPERATIONS.md: production deploys, rollback, backups, monitoring and alerts.
• SCALING.md: capacity settings and running several API and worker nodes.

What runs

Run it with Docker Compose

git clone https://github.com/sadeghianme/Crawler.git
cd Crawler
cp .env.example .env   # then set JWT_SECRET and SECRETS_KEY to long random strings
docker compose up --build

Open http://localhost:8088 and sign up. The API is also reachable directly at http://localhost:8080. Without SMTP settings, verification and password-reset emails are written to the server log (docker compose logs server). Add crawl workers with docker compose up --scale worker=4.

For production, the repository layers docker-compose.prod.yml over the base file (docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d). It adds memory limits and restart policies, removes host port bindings and expects a reverse proxy on an external web_proxy network. It is the hosted service's own configuration (for example, APP_URL is set to the hosted domain), so review it before reusing it. OPERATIONS.md describes the deploy and rollback scripts built around it.

Configuration

Settings are environment variables, documented in .env.example. The ones most deployments need:

A super admin can also set the model, the search engines and their keys in the dashboard under Global Settings; values stored there take precedence over the environment.

Point the SDKs at your server

Set the base URL to your server's API root, including /api when requests go through the web container (for the default Compose setup, http://localhost:8088/api). The API container itself serves the same routes without the prefix, at http://localhost:8080.

export XDIVE_BASE_URL="https://your-host/api"   # read by both SDKs
curl "$XDIVE_BASE_URL/whoami" -H "X-API-Key: $XDIVE_API_KEY"

from xdive import XDive
 
xdive = XDive(base_url="https://your-host/api")  # or set XDIVE_BASE_URL

import { XDive } from "xdive";
 
const xdive = new XDive({ baseUrl: "https://your-host/api" }); // or set XDIVE_BASE_URL

API keys are issued by your server, in its dashboard, exactly as on the hosted service. The stdio MCP server in the repository (cmd/mcp) reads its base URL from CRAWLER_API_URL, not from XDIVE_BASE_URL, and its key from CRAWLER_API_KEY.

Egress proxy

Some sites refuse all connections from data-centre addresses, which is common for government and banking sites. Set EGRESS_PROXY_URL to an HTTP (CONNECT) or SOCKS5 proxy, typically a residential proxy with country targeting in the username:

EGRESS_PROXY_URL=http://USER-country-in:[email protected]:7000
EGRESS_PROXY_DOMAINS=boi.gov.in,bankofindia.bank.in   # optional: always proxy these

Configuring the proxy does not let anyone use it yet. Proxy access is off for every organization, existing and new, until a super admin turns it on under Global Settings → Organizations, where each organization also gets a monthly cap of proxied pages (200 by default, 0 to allow none, at most 100,000) and shows this month's use. The same settings are available as PATCH /admin/orgs/{id}/proxy with {"enabled": true, "monthly_pages": 500}, and every change is written to the organization's audit log. Only scrapes, crawls and batch scrapes use the proxy; search, research, extract, map and llms.txt never do.

For an enabled organization, fetches go direct first and use the proxy only when the site cannot be reached or stays blocked; domains in EGRESS_PROXY_DOMAINS always use it. Once the organization's cap is reached, its fetches go direct only until the next calendar month. The SSRF guard still applies, so the proxy cannot be used to reach private addresses. Proxied pages report fetch_route: "proxy". A site that blocks data-centre networks returns 403 with PROXY_NOT_ENABLED for an organization without access (or past its cap), and 403 with SITE_RESTRICTED when the proxy provider refuses the destination.

Access control

By default anyone can sign up and use the server immediately. Three variables change that:

• SUPER_ADMIN_EMAILS: a comma-separated list of addresses that become platform super admins. An address is promoted only after its owner verifies it, at verification or at sign-in; until then, signing in returns 403 EMAIL_NOT_VERIFIED and emails a new link. Super admins see Global Settings in the dashboard.
• Invite-only sign-up: super admins turn it on under Global Settings → Access requests → Sign-up policy, and keep an allowlist of addresses and @domain entries there. Allowlisted addresses are approved as soon as they sign up; everyone else is held as an access request, as with REGISTRATION_APPROVAL=required. Changes apply without a restart. REGISTRATION_ALLOWED_EMAILS adds entries from the environment and also turns invite-only on.
• REGISTRATION_APPROVAL=required: new sign-ups are held for review. Sign-up answers 202 with "status": "pending_approval", the addresses in SUPER_ADMIN_EMAILS are emailed, and super admins approve or reject requests under Global Settings → Access requests. The user is emailed when approved; until then signing in returns 403 ACCOUNT_PENDING_APPROVAL. Rejecting an account ends its sessions and OAuth grants at once and suspends the organizations it is the only member of.

Accounts listed in SUPER_ADMIN_EMAILS are approved when their address is verified, so they are never left waiting for another administrator.