XDiveDocs
Sign up
View .md

LlamaIndex

XDiveReader loads a web page, or every page of a crawled site, into LlamaIndex Documents with the page's Markdown as text. Use it to build an index over documentation, a knowledge base or any site, without writing fetching or HTML-cleaning code.

Install

bash
pip install "xdive[llama-index]"   # installs llama-index-core

The reader is part of the Python SDK and reads XDIVE_API_KEY (and XDIVE_BASE_URL) like the client. Scrape mode needs a key with the scrape scope, crawl mode the crawl scope.

Load a page or a site

python
from xdive.llama_index import XDiveReader
 
reader = XDiveReader()
 
documents = reader.load_data("https://example.com/pricing")   # one page
documents = reader.load_data(
    "https://docs.example.com",
    mode="crawl",
    max_pages=100,
    include_paths=["^/guides/"],
)
 
for document in documents:
    print(document.metadata["url"], document.metadata["title"], len(document.text))

XDiveReader(client=None) takes an optional XDive client; by default it creates XDive() from the environment. Pass your own client for a self-hosted server or other settings:

python
from xdive import XDive
from xdive.llama_index import XDiveReader
 
reader = XDiveReader(XDive(api_key="ck_live_…", base_url="https://your-host/api"))

load_data(url, mode="scrape", **options) returns a list[Document]:

ArgumentDescription
urlThe page to scrape, or the seed URL of the crawl.
mode"scrape" loads the one URL. "crawl" crawls from it and loads every page that did not fail. Any other value raises ValueError.
**optionsPassed to XDive.scrape or XDive.crawl: crawl parameters such as max_pages, max_depth, include_paths and exclude_paths (see Crawl), and the client's timeout and poll_interval for crawls.

Each document's text is the page's Markdown, and its metadata is url and title. Markdown is always requested, so do not pass formats.

Crawl mode waits until the crawl finishes before returning; the client's default wait is 600 seconds (timeout), after which CrawlTimeout is raised and the crawl keeps running on the server. API errors raise the SDK's exceptions.

Build an index

python
from llama_index.core import VectorStoreIndex
 
from xdive.llama_index import XDiveReader
 
documents = XDiveReader().load_data("https://docs.example.com", mode="crawl", max_pages=200)
index = VectorStoreIndex.from_documents(documents)
 
answer = index.as_query_engine().query("How do I rotate an API key?")
print(answer)

VectorStoreIndex embeds the documents and answers queries with the embedding model and LLM configured in LlamaIndex's Settings; those calls are billed by their providers, not by XDive.

Usage

Scrape mode counts one page. Crawl mode counts every page in the crawl's results, including pages that failed and were left out of the documents. To see which pages failed, crawl with the client directly and call crawl_errors with the crawl id. See Usage and billing.