Ruby SDK

Official Ruby client for the XDive web-data API: scrape pages to Markdown, crawl sites, search the web and get cited research answers.

• Ruby 3.0+, standard library only (net/http, json, securerandom, uri), no Rails dependency (works in Rails).
• Thread-safe: one client can be shared by every thread; each call uses its own connection.
• Automatic retries with Retry-After, and an Idempotency-Key on every POST so a retried call never runs (or bills) twice.

Install

gem install xdive

or in your Gemfile:

gem "xdive"

Quickstart

require "xdive"
 
client = XDive::Client.new(api_key: ENV.fetch("XDIVE_API_KEY"))
# or simply: client = XDive.client   (reads XDIVE_API_KEY / XDIVE_BASE_URL)
 
# Scrape one page
page = client.scrape("https://example.com", formats: %w[markdown links])
puts page.title
puts page.markdown          # also XDive.page_markdown(page)
 
# Crawl a site and wait for it (start_crawl + wait_for_crawl)
record = client.crawl("https://docs.example.com", limit: 50, formats: ["markdown"])
record.results.each { |p| puts "#{p.url}: #{p.markdown.length} chars" } if record.done?
 
# Search the web
hits = client.search("ruby 4.0 release notes", limit: 5, include_answer: true)
hits["results"].each { |r| puts "#{r['title']} - #{r['url']}" }
 
# Research: a cited answer
result = client.research("What changed in Ruby 4.0?", depth: "quick")
puts result.answer
result.citations.each { |c| puts "[#{c['number']}] #{c['url']}" }

All methods

Results are plain string-keyed Hashes; Page, CrawlRecord and ResearchResult are thin wrappers that expose the complete response as #raw / #to_h and accept result[:key]. Request fields the gem does not model pass through as extra keyword arguments or a trailing Hash: client.scrape(url, only_main_content: true), client.scrape(url, { "wait_for" => 500 }). nil values are dropped. formats: takes names ("markdown", :links) or option hashes ({ type: "json", schema: {...} }).

Rails

# config/initializers/xdive.rb
XDive.configure(
  api_key: Rails.application.credentials.dig(:xdive, :api_key),
  timeout: 60,
  max_retries: 3
)

Then use the shared, thread-safe client anywhere: XDive.client.scrape("https://example.com").markdown. For long crawls prefer a background job (start_crawl in the request, wait_for_crawl or get_crawl in the job).

Errors

Every failed call raises a subclass of XDive::Error:

begin
  client.scrape("https://example.com")
rescue XDive::RateLimitedError => e
  sleep e.retry_after            # still throttled after max_retries
rescue XDive::QuotaExceededError
  # HTTP 402: plan quota used up
rescue XDive::CrawlTimeoutError => e
  e.crawl_id                     # wait_for_crawl gave up; the crawl keeps running
rescue XDive::ConnectionError
  # DNS / refused / timeout / TLS, after retries (see #cause)
rescue XDive::Error => e
  warn "#{e.status} #{e.code}: #{e.message} (request #{e.request_id}) #{e.docs_url}"
  retry if e.retryable?
end

#status, #code, #retryable?, #request_id (the X-Request-ID header) and #docs_url come from the API's error body {"success":false,"code":"..","error":"..","retryable":bool,"docs_url":".."}; older bodies with only error leave #code empty.

Retries and idempotency

• 429, 503 with code SERVER_BUSY and 409 with code IDEMPOTENCY_KEY_IN_PROGRESS are retried after Retry-After seconds (default 1, fractional allowed), up to max_retries (default 3), then RateLimitedError is raised. Other statuses are never retried.
• Transport errors and timeouts on GET and POST are retried with a 1s, 2s, 4s... backoff (cap 8s), then ConnectionError is raised.
• Every POST carries an Idempotency-Key (a UUID generated once per call and reused by every retry of that call). Pass idempotency_key: "my-key" to make a call safe to repeat across processes.
• wait_for_crawl sends If-None-Match, so an unchanged crawl costs a bodiless 304, and follows the server's Retry-After.

Configuration

See also

• Errors and limits, for every error code and the rate limits.
• Python, JavaScript and Go SDKs have the same methods.