# Ruby SDK

Official Ruby client for the [XDive](https://xdive.dev) web-data API: scrape pages to Markdown, crawl sites, search the web and get cited research answers.

- Ruby 3.0+, standard library only (`net/http`, `json`, `securerandom`, `uri`), no Rails dependency (works in Rails).
- Thread-safe: one client can be shared by every thread; each call uses its own connection.
- Automatic retries with `Retry-After`, and an `Idempotency-Key` on every POST so a retried call never runs (or bills) twice.

## Install

```sh
gem install xdive
```

or in your `Gemfile`:

```ruby
gem "xdive"
```

## Quickstart

```ruby
require "xdive"

client = XDive::Client.new(api_key: ENV.fetch("XDIVE_API_KEY"))
# or simply: client = XDive.client   (reads XDIVE_API_KEY / XDIVE_BASE_URL)

# Scrape one page
page = client.scrape("https://example.com", formats: %w[markdown links])
puts page.title
puts page.markdown          # also XDive.page_markdown(page)

# Crawl a site and wait for it (start_crawl + wait_for_crawl)
record = client.crawl("https://docs.example.com", limit: 50, formats: ["markdown"])
record.results.each { |p| puts "#{p.url}: #{p.markdown.length} chars" } if record.done?

# Search the web
hits = client.search("ruby 4.0 release notes", limit: 5, include_answer: true)
hits["results"].each { |r| puts "#{r['title']} - #{r['url']}" }

# Research: a cited answer
result = client.research("What changed in Ruby 4.0?", depth: "quick")
puts result.answer
result.citations.each { |c| puts "[#{c['number']}] #{c['url']}" }
```

### All methods

| Method | Returns |
| --- | --- |
| `scrape(url, formats:, **opts)` | `XDive::Page` |
| `start_crawl(url, formats:, **opts)` / `crawl(url, timeout: 600, poll_interval: 2, ...)` | job id / `XDive::CrawlRecord` |
| `get_crawl(id)`, `wait_for_crawl(id, timeout:, poll_interval:)` | `XDive::CrawlRecord` (`#done?`, `#results`) |
| `cancel_crawl(id)`, `crawl_errors(id)` | `nil`, `Hash` |
| `start_batch_scrape(urls, ...)` / `batch_scrape(urls, ...)` | job id / `XDive::CrawlRecord` |
| `map(url, search:, limit:, max_depth:)` | `Array<String>` |
| `search(query, **opts)` | `Hash` |
| `research(query, depth:, output:, schema:, no_cache:)` | `XDive::ResearchResult` (`#answer`, `#citations`) |
| `extract(urls, prompt:, schema:)`, `llms_txt(url, max_urls:, full_text:)`, `whoami` | `Hash` |

Results are plain string-keyed Hashes; `Page`, `CrawlRecord` and `ResearchResult` are thin wrappers that expose the complete response as `#raw` / `#to_h` and accept `result[:key]`. Request fields the gem does not model pass through as extra keyword arguments or a trailing Hash: `client.scrape(url, only_main_content: true)`, `client.scrape(url, { "wait_for" => 500 })`. `nil` values are dropped. `formats:` takes names (`"markdown"`, `:links`) or option hashes (`{ type: "json", schema: {...} }`).

## Rails

```ruby
# config/initializers/xdive.rb
XDive.configure(
  api_key: Rails.application.credentials.dig(:xdive, :api_key),
  timeout: 60,
  max_retries: 3
)
```

Then use the shared, thread-safe client anywhere: `XDive.client.scrape("https://example.com").markdown`. For long crawls prefer a background job (`start_crawl` in the request, `wait_for_crawl` or `get_crawl` in the job).

## Errors

Every failed call raises a subclass of `XDive::Error`:

```ruby
begin
  client.scrape("https://example.com")
rescue XDive::RateLimitedError => e
  sleep e.retry_after            # still throttled after max_retries
rescue XDive::QuotaExceededError
  # HTTP 402: plan quota used up
rescue XDive::CrawlTimeoutError => e
  e.crawl_id                     # wait_for_crawl gave up; the crawl keeps running
rescue XDive::ConnectionError
  # DNS / refused / timeout / TLS, after retries (see #cause)
rescue XDive::Error => e
  warn "#{e.status} #{e.code}: #{e.message} (request #{e.request_id}) #{e.docs_url}"
  retry if e.retryable?
end
```

`#status`, `#code`, `#retryable?`, `#request_id` (the `X-Request-ID` header) and `#docs_url` come from the API's error body `{"success":false,"code":"..","error":"..","retryable":bool,"docs_url":".."}`; older bodies with only `error` leave `#code` empty.

## Retries and idempotency

- `429`, `503` with code `SERVER_BUSY` and `409` with code `IDEMPOTENCY_KEY_IN_PROGRESS` are retried after `Retry-After` seconds (default 1, fractional allowed), up to `max_retries` (default 3), then `RateLimitedError` is raised. Other statuses are never retried.
- Transport errors and timeouts on GET and POST are retried with a 1s, 2s, 4s... backoff (cap 8s), then `ConnectionError` is raised.
- Every POST carries an `Idempotency-Key` (a UUID generated once per call and reused by every retry of that call). Pass `idempotency_key: "my-key"` to make a call safe to repeat across processes.
- `wait_for_crawl` sends `If-None-Match`, so an unchanged crawl costs a bodiless `304`, and follows the server's `Retry-After`.

## Configuration

| Option | Env var (fallback) | Default |
| --- | --- | --- |
| `api_key:` | `XDIVE_API_KEY` (`CRAWLER_API_KEY`) | required |
| `base_url:` | `XDIVE_BASE_URL` (`CRAWLER_BASE_URL`) | `XDive::DEFAULT_BASE_URL` |
| `org_id:` | none | discovered once via `GET /whoami`, cached |
| `max_retries:` | none | `3` |
| `timeout:` | none | `120` seconds (connect, read and write) |

## See also

- [Errors and limits](https://xdive.dev/docs/errors.md), for every error code and the rate limits.
- [Python](https://xdive.dev/docs/sdk-python.md), [JavaScript](https://xdive.dev/docs/sdk-javascript.md) and [Go](https://xdive.dev/docs/sdk-go.md) SDKs have the same methods.

---

Source: https://xdive.dev/docs/sdk-ruby · All docs for agents: https://xdive.dev/llms.txt