# Map

List a site's URLs quickly, without converting any content. Map reads the site's sitemaps and runs a links-only crawl, then returns the same-site URLs it found, optionally filtered by a search term. Use it to plan a crawl, pick pages for a [batch scrape](https://xdive.dev/docs/batch-scrape.md), or find where a topic lives on a site.

`POST https://xdive.dev/api/orgs/{id}/map`

Requires an API key with the `crawl` scope. Map is synchronous and returns within 45 seconds.

**cURL**

```bash
curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/map" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "search": "webhook", "limit": 100}'
```

**Python**

```python
from xdive import XDive

xdive = XDive()  # reads XDIVE_API_KEY
urls = xdive.map("https://docs.example.com", search="webhook", limit=100)
print(len(urls), urls[:5])
```

**JavaScript**

```ts
import { XDive } from "xdive";

const xdive = new XDive(); // reads XDIVE_API_KEY
const urls = await xdive.map("https://docs.example.com", { search: "webhook", limit: 100 });
console.log(urls.length, urls.slice(0, 5));
```

The SDK methods return the `urls` array.

## Parameters

| Field | Type | Default | Description |
|---|---|---|---|
| `url` | string | required | The page to start from. Its registrable domain is the site that is mapped. |
| `search` | string | none | Keep only URLs that contain this text, case-insensitively. Pages that do not match are still traversed, so matches deep in the site are found. |
| `limit` | integer | `5000` | Maximum URLs to return. Values above 5000, and `0`, mean 5000. |
| `max_depth` | integer | `2` | Link hops for the links-only crawl. |

## How URLs are found

1. **Sitemaps.** The `Sitemap` entries in the site's robots.txt are read, or `/sitemap.xml` when there are none. Sitemap indexes are followed.
2. **Links.** A links-only crawl starts at `url` and follows links up to `max_depth` hops, fetching at most 200 pages. Every same-site link on those pages is collected, including links to pages that were not fetched, so the result can be much larger than 200.

Only `http` and `https` URLs on the seed's registrable domain are returned (for `docs.example.com`, URLs on `www.example.com` count too). Duplicates are removed. Pages that robots.txt disallows are not fetched. Sitemap URLs come first, in sitemap order, followed by the crawl's discoveries.

Map is a discovery tool: it does not check that each URL still resolves, and it returns URLs in the form the sitemap or link used.

## Response

```json
{
  "count": 3,
  "urls": [
    "https://docs.example.com/guides/webhooks",
    "https://docs.example.com/reference/webhook-events",
    "https://docs.example.com/blog/signed-webhooks"
  ]
}
```

| Field | Type | Description |
|---|---|---|
| `count` | integer | Number of URLs returned. |
| `urls` | array of strings | The URLs. |

## Usage

Map does not count pages against your quota, but it is refused with `402` once your monthly page quota is used up. It counts against your plan's request rate limit and concurrency limit like other synchronous endpoints.

## Errors

| Status | Code | When |
|---|---|---|
| `400` | | `url` is missing or not an absolute `http` or `https` URL, or the body is not valid JSON. |
| `402` | | The monthly page quota is used up. |
| `403` | | The key lacks the `crawl` scope. |
| `429` | | Your plan's request rate limit was exceeded. Retry after `Retry-After`. |
| `429` | `CONCURRENCY_LIMIT` | Too many concurrent requests for your plan. Retry after `Retry-After`. |
| `503` | `SERVER_BUSY` | The server is at capacity. Retry after `Retry-After`. |

See [Errors and limits](https://xdive.dev/docs/errors.md) for authentication errors and rate limits.

## Related

- [Crawl](https://xdive.dev/docs/crawl.md) to fetch and convert the pages.
- [Batch scrape](https://xdive.dev/docs/batch-scrape.md) to scrape a chosen subset.
- [llms.txt](https://xdive.dev/docs/llms-txt.md) to build an index of a site for language models.

---

Source: https://xdive.dev/docs/map · All docs for agents: https://xdive.dev/llms.txt