XDiveDocs
Sign up
View .md

Vite plugin

vite-plugin-xdive turns web pages into LLM-ready Markdown while you build, so your app can import them like any other module. Use it to ship docs, pricing or policy pages that are kept in step with a source site, or to generate an llms.txt for your own site. It uses your API key and the scrape endpoint, and caches every page on disk.

Install

sh
npm install -D vite-plugin-xdive
pnpm add -D vite-plugin-xdive
yarn add -D vite-plugin-xdive
bun add -d vite-plugin-xdive

The plugin needs Vite 5, 6 or 7 and Node 18 or newer. Set your API key in the environment:

sh
export XDIVE_API_KEY=ck_live_...

Usage

ts
// vite.config.ts
import { defineConfig } from "vite";
import xdive from "vite-plugin-xdive";
 
export default defineConfig({
  plugins: [
    xdive({
      pages: [
        { id: "pricing", url: "https://example.com/pricing" },
        { url: "https://example.com/docs/intro" }, // id: example-com-docs-intro
      ],
      llmsTxt: { title: "Example", summary: "Docs and pricing for Example." },
    }),
  ],
});

For TypeScript, add once (for example in src/env.d.ts):

ts
/// <reference types="vite-plugin-xdive/client" />

React:

tsx
import { pages } from "virtual:xdive";
 
export function Pricing() {
  return <pre>{pages.pricing.markdown}</pre>;
}

Vue:

vue
<script setup lang="ts">
import { pages } from "virtual:xdive";
const { title, markdown } = pages.pricing;
</script>
<template><h1>{{ title }}</h1><pre>{{ markdown }}</pre></template>

Each page is { url, title, markdown, fetchedAt }. pages is also the default export.

Options

OptionTypeDefaultDescription
pages{ id?, url, formats?, maxAgeMs? }[]requiredPages to fetch. id defaults to a slug of the URL; markdown is always requested; a cached copy younger than maxAgeMs is used without calling the API.
apiKeystringenv XDIVE_API_KEY, then CRAWLER_API_KEYSent as X-API-Key. Never written to the cache or logs.
baseUrlstringenv XDIVE_BASE_URL, then CRAWLER_BASE_URL, then the public APIAPI origin including /api.
cacheDirstringnode_modules/.cache/xdiveRelative to the Vite root.
offlinebooleanfalseUse only the cache; no API calls, no key needed. A page with no cached copy is an error in build and omitted with a warning in dev.
failOnErrorbooleanunsetSee below.
llmsTxt{ title, summary?, outFile? }offEmit llms.txt (llmstxt.org format) and <id>.md for every page into the build output. outFile defaults to llms.txt.

.env files are read too (XDIVE_API_KEY in .env.local works).

Caching and failures

One JSON file per page is kept in the cache directory.

  • vite build always refreshes every page (except pages whose cached copy is younger than maxAgeMs).
  • vite dev serves the cached copy instantly and refreshes it in the background; if the content changed, the virtual module is invalidated and the page reloads. A page with no cache is fetched before the first load.
  • If a fetch fails, failOnError decides:
    • unset (default): the cached copy is used with a warning. With no cached copy, the build fails and vite dev omits the page with a warning.
    • false: never fails; cached copy with a warning, else the page is omitted with a warning.
    • true: any failed page fails the build or dev start; the cache is not used as a fallback.
  • A missing API key always fails (unless offline), naming XDIVE_API_KEY and CRAWLER_API_KEY.
  • Quota errors (402) and other HTTP errors include the API's message and code.
  • 429 and 503 SERVER_BUSY are retried honoring Retry-After, reusing one Idempotency-Key so a retry never runs or bills a request twice.

CI

Store the key as a secret and expose it as XDIVE_API_KEY:

yaml
- run: pnpm build
  env:
    XDIVE_API_KEY: ${{ secrets.XDIVE_API_KEY }}

Cache node_modules/.cache/xdive between runs so a transient API outage does not break the build.

See also