Map

List a site's URLs quickly, without converting any content. Map reads the site's sitemaps and runs a links-only crawl, then returns the same-site URLs it found, optionally filtered by a search term. Use it to plan a crawl, pick pages for a batch scrape, or find where a topic lives on a site.

Requires an API key with the crawl scope. Map is synchronous and returns within 45 seconds.

curl -X POST "https://xdive.dev/api/orgs/$ORG_ID/map" \
  -H "X-API-Key: $XDIVE_API_KEY" -H "Content-Type: application/json" \
  -d '{"url": "https://docs.example.com", "search": "webhook", "limit": 100}'

from xdive import XDive
 
xdive = XDive()  # reads XDIVE_API_KEY
urls = xdive.map("https://docs.example.com", search="webhook", limit=100)
print(len(urls), urls[:5])

import { XDive } from "xdive";
 
const xdive = new XDive(); // reads XDIVE_API_KEY
const urls = await xdive.map("https://docs.example.com", { search: "webhook", limit: 100 });
console.log(urls.length, urls.slice(0, 5));

The SDK methods return the urls array.

Parameters

How URLs are found

1. Sitemaps. The Sitemap entries in the site's robots.txt are read, or /sitemap.xml when there are none. Sitemap indexes are followed.
2. Links. A links-only crawl starts at url and follows links up to max_depth hops, fetching at most 200 pages. Every same-site link on those pages is collected, including links to pages that were not fetched, so the result can be much larger than 200.

Only http and https URLs on the seed's registrable domain are returned (for docs.example.com, URLs on www.example.com count too). Duplicates are removed. Pages that robots.txt disallows are not fetched. Sitemap URLs come first, in sitemap order, followed by the crawl's discoveries.

Map is a discovery tool: it does not check that each URL still resolves, and it returns URLs in the form the sitemap or link used.

Response

{
  "count": 3,
  "urls": [
    "https://docs.example.com/guides/webhooks",
    "https://docs.example.com/reference/webhook-events",
    "https://docs.example.com/blog/signed-webhooks"
  ]
}

Usage

Map does not count pages against your quota, but it is refused with 402 once your monthly page quota is used up. It counts against your plan's request rate limit and concurrency limit like other synchronous endpoints.

Errors

See Errors and limits for authentication errors and rate limits.

Related

• Crawl to fetch and convert the pages.
• Batch scrape to scrape a chosen subset.
• llms.txt to build an index of a site for language models.