Firecrawl

Firecrawl is developed by Mendable, Inc. The core API server and Playwright scraping service are built in TypeScript, orchestrated via Docker Compose (API + Worker + Redis + PostgreSQL + Playwright microservice), with async tasks scheduled through Bull Queue. The core is open-sourced under AGPL-3.0, while SDKs and some UI components use MIT.

The project exposes eight API endpoints: Search (search and return page content), Scrape (single URL to Markdown/JSON/screenshot), Crawl (full-site crawling), Map (site URL discovery), Batch Scrape (async bulk scraping), Parse (PDF/DOCX parsing), Interact (AI-driven page interaction then extraction), and Agent (autonomous data collection with structured Schema output). It claims 96% web coverage with P95 latency of 3.4s, automatically handling proxy rotation, rate limiting, and other anti-scraping complexities.

Two deployment modes are available: cloud SaaS (full feature set including Fire-engine advanced anti-blocking and Agent/Browser endpoints) and self-hosted (feature subset, but with experimental Ollama local LLM support and SearXNG as an alternative search backend). The AI layer supports OpenAI API, Ollama, and any OpenAI-compatible endpoint.

For agent integration, it provides a CLI Skill for one-click installation and a standard MCP Server configuration. It has been integrated with agents like Claude Code, Antigravity, and OpenCode, as well as automation platforms like Zapier and n8n. Official SDKs are available for Python, Node.js, Rust, Elixir, and Java; a Go SDK is community-maintained.

SDK Installation#

Python:

pip install firecrawl-py

Node.js:

npm install @mendable/firecrawl-js

Rust:

[dependencies]
firecrawl = "2"

Self-Hosted Deployment#

Prerequisite: Docker. Create a .env config file, then run:

docker compose build
docker compose up

The service defaults to http://localhost:3002. Kubernetes Helm Chart deployment is also supported.

Quick Start#

from firecrawl import Firecrawl
app = Firecrawl(api_key="fc-YOUR_API_KEY")
doc = app.scrape("https://firecrawl.dev", formats=["markdown"])
print(doc.markdown)

MCP Server Configuration#

{
  "mcpServers": {
    "firecrawl-mcp": {
      "command": "npx",
      "args": ["-y", "firecrawl-mcp"],
      "env": { "FIRECRAWL_API_KEY": "fc-YOUR_API_KEY" }
    }
  }
}

Cloud vs Self-Hosted Feature Comparison#