# LlamaIndex

`XDiveReader` loads a web page, or every page of a crawled site, into LlamaIndex `Document`s with the page's Markdown as text. Use it to build an index over documentation, a knowledge base or any site, without writing fetching or HTML-cleaning code.

## Install

```bash
pip install "xdive[llama-index]"   # installs llama-index-core
```

The reader is part of the Python SDK and reads `XDIVE_API_KEY` (and `XDIVE_BASE_URL`) like the client. Scrape mode needs a key with the `scrape` scope, crawl mode the `crawl` scope.

## Load a page or a site

```python
from xdive.llama_index import XDiveReader

reader = XDiveReader()

documents = reader.load_data("https://example.com/pricing")   # one page
documents = reader.load_data(
    "https://docs.example.com",
    mode="crawl",
    max_pages=100,
    include_paths=["^/guides/"],
)

for document in documents:
    print(document.metadata["url"], document.metadata["title"], len(document.text))
```

`XDiveReader(client=None)` takes an optional `XDive` client; by default it creates `XDive()` from the environment. Pass your own client for a self-hosted server or other settings:

```python
from xdive import XDive
from xdive.llama_index import XDiveReader

reader = XDiveReader(XDive(api_key="ck_live_…", base_url="https://your-host/api"))
```

`load_data(url, mode="scrape", **options)` returns a `list[Document]`:

| Argument | Description |
|---|---|
| `url` | The page to scrape, or the seed URL of the crawl. |
| `mode` | `"scrape"` loads the one URL. `"crawl"` crawls from it and loads every page that did not fail. Any other value raises `ValueError`. |
| `**options` | Passed to `XDive.scrape` or `XDive.crawl`: crawl parameters such as `max_pages`, `max_depth`, `include_paths` and `exclude_paths` (see [Crawl](https://xdive.dev/docs/crawl.md)), and the client's `timeout` and `poll_interval` for crawls. |

Each document's `text` is the page's Markdown, and its `metadata` is `url` and `title`. Markdown is always requested, so do not pass `formats`.

Crawl mode waits until the crawl finishes before returning; the client's default wait is 600 seconds (`timeout`), after which `CrawlTimeout` is raised and the crawl keeps running on the server. API errors raise the [SDK's exceptions](https://xdive.dev/docs/sdk-python.md#errors).

## Build an index

```python
from llama_index.core import VectorStoreIndex

from xdive.llama_index import XDiveReader

documents = XDiveReader().load_data("https://docs.example.com", mode="crawl", max_pages=200)
index = VectorStoreIndex.from_documents(documents)

answer = index.as_query_engine().query("How do I rotate an API key?")
print(answer)
```

`VectorStoreIndex` embeds the documents and answers queries with the embedding model and LLM configured in LlamaIndex's `Settings`; those calls are billed by their providers, not by XDive.

## Usage

Scrape mode counts one page. Crawl mode counts every page in the crawl's results, including pages that failed and were left out of the documents. To see which pages failed, crawl with the client directly and call `crawl_errors` with the crawl id. See [Usage and billing](https://xdive.dev/docs/usage.md).

## Related

- [LangChain](https://xdive.dev/docs/langchain.md)
- [Python SDK](https://xdive.dev/docs/sdk-python.md)
- [Crawl](https://xdive.dev/docs/crawl.md)

---

Source: https://xdive.dev/docs/llamaindex · All docs for agents: https://xdive.dev/llms.txt