XDiveDocs
Sign up
View .md

LangChain

Both SDKs ship LangChain integrations: three agent tools (search, scrape and research) for LangChain in Python and LangChain.js, and a document loader for Python. The tools return compact text built for a model's context window: the search tool returns the passages of each result that match the query, not whole pages.

Install

bash
pip install "xdive[langchain]"   # installs langchain-core

In Python the tools live in xdive.langchain; in JavaScript they are imported from xdive/langchain. @langchain/core and zod are optional peer dependencies of the npm package, needed only for this entry point.

The tools read XDIVE_API_KEY (and XDIVE_BASE_URL) like the SDK clients. The key needs the search scope for the search and research tools and the scrape scope for the scrape tool; the loader's crawl mode needs crawl.

Tools

ClassTool nameInputReturns
XDiveSearchToolxdive_searchquery, limit (default 5, 1 to 20), freshness (pd, pw, pm or py, optional)JSON array of results, each with its relevant passages.
XDiveScrapeToolxdive_scrapeurlThe page's Markdown, cut at 60,000 characters.
XDiveResearchToolxdive_researchquery, depth (quick or deep, default quick)JSON object with a cited answer and its citations.

The tool names and descriptions are what the model sees when it decides which tool to call. The search tool's description points the model to xdive_research when it needs a synthesized answer.

Calls search with search_depth: "advanced" and auto_parameters: true: the API reads the top web results and keeps the excerpts most relevant to the query, and infers news sources and a time range from the wording when the query asks for recent information. The tool returns a JSON array:

json
[
  {
    "title": "What's new in Python packaging",
    "url": "https://packaging.example.org/news/license-metadata",
    "snippet": "License-Expression and License-File fields replace the old classifiers…",
    "passages": [
      "PEP 639 adds the License-Expression field, which holds an SPDX expression such as MIT OR Apache-2.0…",
      "Build backends copy the files named by license-files into the .dist-info/licenses directory…"
    ],
    "relevance": 11.82
  }
]
FieldDescription
title, urlThe search result.
snippetThe search engine's description of the result.
passagesUp to 3 excerpts of the page that best match the query, best first. Empty for results the API did not read or found nothing relevant in.
relevanceThe score of the best passage: BM25 relevance to the query, not normalized; higher is more relevant. Absent (JavaScript) or null (Python) when passages is empty.

The API reads at most 8 results per call, keeping the search engine's order. A call counts as one search plus one page for each result read.

In Python, limit is clamped to 1 to 20; in JavaScript the input schema rejects values outside that range and freshness must be one of the four values.

xdive_scrape

Calls scrape with the Markdown format and returns the Markdown as plain text, cut at 60,000 characters so one long page cannot fill the context. In JavaScript the input schema requires a valid URL. A call counts as one page.

xdive_research

Calls research and returns a JSON object:

json
{
  "answer": "PEP 639 replaces license classifiers with an SPDX License-Expression field [1] and records license files in the wheel [2].",
  "confidence": "high",
  "citations": [
    { "n": 1, "url": "https://peps.python.org/pep-0639/", "title": "PEP 639" },
    { "n": 2, "url": "https://packaging.example.org/news/license-metadata", "title": "What's new in Python packaging" }
  ]
}

answer holds the answer text with numbered citation markers; n is the marker's number. quick research takes seconds, deep one to two minutes. A call counts the searches and pages the research used; a result served from the short-lived cache is not counted. See Usage and billing.

Sharing a client

Each tool creates its own client from the environment unless you give it one. To share configuration and the cached organization id, pass a client:

python
from xdive import XDive
from xdive.langchain import XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool
 
xdive = XDive(api_key="ck_live_…", base_url="https://your-host/api")
tools = [
    XDiveSearchTool(client=xdive),
    XDiveScrapeTool(client=xdive),
    XDiveResearchTool(client=xdive),
]

In Python the client is the client keyword argument; in JavaScript it is the constructor's only argument.

Build an agent

The tools work with any LangChain chat model that supports tool calling. This example uses LangChain's create_agent / createAgent with an Anthropic model, which needs langchain and the Anthropic integration installed and ANTHROPIC_API_KEY set:

bash
pip install "xdive[langchain]" langchain langchain-anthropic
python
from langchain.agents import create_agent
 
from xdive import XDive
from xdive.langchain import XDiveResearchTool, XDiveScrapeTool, XDiveSearchTool
 
xdive = XDive()  # reads XDIVE_API_KEY
agent = create_agent(
    model="anthropic:claude-opus-5-5",
    tools=[
        XDiveSearchTool(client=xdive),
        XDiveScrapeTool(client=xdive),
        XDiveResearchTool(client=xdive),
    ],
    system_prompt="Answer from the web and cite the URLs you used.",
)
 
result = agent.invoke(
    {"messages": [{"role": "user", "content": "What changed in the latest Node.js LTS release?"}]}
)
print(result["messages"][-1].content)

A typical run calls xdive_search to find sources, reads the passages, and calls xdive_scrape on a result when it needs the whole page. For questions that need several sources, xdive_research returns a cited answer in one call.

The tools raise the SDK's errors (XDiveError and its subclasses in Python, the matching error classes in JavaScript) after the client's own retries. How the agent reacts to a failed tool call depends on your LangChain version's tool-error handling. See Python SDK errors and JavaScript SDK errors.

Document loader

XDiveLoader (Python) loads one page or a whole site as LangChain Documents, for indexing in a vector store or passing to a chain.

python
from xdive.langchain import XDiveLoader
 
page = XDiveLoader("https://example.com/pricing").load()
site = XDiveLoader("https://docs.example.com", mode="crawl", max_pages=100, max_depth=3).load()
 
for doc in site:
    print(doc.metadata["source"], len(doc.page_content))

XDiveLoader(url, *, mode="scrape", client=None, **options)

ArgumentDescription
urlThe page to scrape, or the seed URL of the crawl.
mode"scrape" loads the one URL; "crawl" crawls from it and loads every page. Any other value raises ValueError.
clientAn XDive client to use. Defaults to XDive(), which reads the environment.
**optionsPassed to XDive.scrape or XDive.crawl: crawl parameters such as max_pages, max_depth, include_paths and exclude_paths, and the client's timeout and poll_interval for crawls.

load() returns a list; lazy_load() yields the documents one by one. Each document's page_content is the page's Markdown. Its metadata holds source (the page URL) and, when the page has them, url, title, description, lang and fetched_at. In crawl mode, pages that failed are skipped.

A crawl waits for the whole crawl to finish before the first document is returned; the client's default wait is 600 seconds (timeout), after which CrawlTimeout is raised.

The JavaScript package has no loader class. Build documents from a crawl with pageMarkdown:

ts
import { Document } from "@langchain/core/documents";
import { XDive, pageMarkdown } from "xdive";
 
const xdive = new XDive();
const record = await xdive.crawl("https://docs.example.com", { max_pages: 100 });
const docs = (record.results ?? [])
  .filter((page) => !page.error)
  .map((page) => new Document({ pageContent: pageMarkdown(page), metadata: { source: page.url, title: page.title } }));