CrawlAPI converts web pages into clean markdown or structured data — ready for LangChain, CrewAI, OpenAI Assistants, and any agent framework.
const response = await fetch('https://crawlapi.net/v1/scrape', { method: 'POST', headers: { 'X-RapidAPI-Key': 'YOUR_KEY', 'Content-Type': 'application/json' }, body: JSON.stringify({ url: 'https://example.com', formats: ['markdown'] }) }); // → { success: true, data: { markdown: "# Example Domain\n..." } }
Ships with OpenAPI spec + ready-made tool manifests for LangChain, CrewAI, and OpenAI Assistants. Drop in, done.
HTTP + Cheerio for static pages, Puppeteer fallback for JS-rendered SPAs. Redis caching keeps repeat calls instant.
Mozilla Readability strips ads and boilerplate. You get the content, not the noise.
Send up to 10 URLs in a single request. Parallel processing, single API call.
Pass a query, get back scraped content from the top results. One call, fully researched.
Firecrawl starts at $19/mo. ScrapingBee at $49/mo. CrawlAPI Pro is $9.99/mo.
Scrape a single URL → markdown, plain text, or both. Supports waitFor (JS rendering) and custom timeout.
Scrape up to 10 URLs in parallel. Returns an array of results with per-URL success/error status.
Pass a search query, get back scraped content from the top results. Perfect for research agents.