# CrawlAPI

> Web scraping and search API designed for AI agents, LLM pipelines, and autonomous workflows.

## What CrawlAPI Does

CrawlAPI converts any URL into clean, structured data optimised for LLM consumption.
It handles JavaScript rendering, removes boilerplate, and returns markdown or structured JSON — ready to drop straight into your prompts.

## Base URL

https://crawlapi.net

## Authentication

All API endpoints require an `X-RapidAPI-Key` header (via RapidAPI) or a direct API key header.

## Endpoints

### POST /v1/scrape
Scrape a single URL. Returns markdown, HTML, or structured metadata.

**Request:**
```json
{
  "url": "https://example.com",
  "formats": ["markdown"],
  "waitFor": 1000,
  "timeout": 30000
}
```

**Response:**
```json
{
  "success": true,
  "data": {
    "markdown": "# Page Title\n\nPage content...",
    "metadata": {
      "title": "Page Title",
      "description": "Meta description",
      "url": "https://example.com"
    }
  }
}
```

### POST /v1/batch
Scrape up to 10 URLs in parallel. Same format options as /v1/scrape.

**Request:**
```json
{
  "urls": ["https://example.com", "https://example.org"],
  "formats": ["markdown"]
}
```

### POST /v1/search
Search the web and get scraped content for the top N results.

**Request:**
```json
{
  "query": "best pizza in New York",
  "num": 5,
  "formats": ["markdown"]
}
```

## Supported Formats

- `markdown` — Clean markdown, ideal for LLM context windows
- `html` — Raw HTML after JavaScript rendering
- `text` — Plain text, no formatting
- `structured` — JSON with title, description, headings, links, images

## Use Cases for AI Agents

- **RAG pipelines:** Fetch and chunk web content on-demand
- **Autonomous research:** Give agents real-time web access
- **Data extraction:** Pull structured data from any public page
- **Search + summarise:** Query the web and process results in one step
- **Tool use:** Drop into any agent framework as a `browse_web` or `fetch_url` tool

## Framework Integration

Works with LangChain, LlamaIndex, CrewAI, AutoGen, n8n, Flowise, and any framework that supports HTTP tool calls.

## RapidAPI Listing

https://rapidapi.com/crawlapi/api/crawlapi

## Landing Page

https://crawlapi.net

## Pricing

Available via RapidAPI. Free tier included.