crawl_site
Crawl an entire site and return a manifest of URL/title/snippet for each page. Tries Firecrawl crawl first, falls back to sitemap parsing, then optional BFS. Full page content is cached in Valkey so follow-up fetchurl calls are zero-cost.
How to use it
crawl_site is exposed by the Searxng MCP MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the crawl_site tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.
Install Searxng MCP
docker run -d -p 11235:11235 unclecode/crawl4ai:0.8.6Other tools in Searxng MCP (6)
Purge the search cache, fetch cache, crawl manifest cache, or all. Useful when researching fast-moving topics where cached results may be stale.
Read-only view of the domain capability database. With hostname: one domain's per-tier success rates and capability flags. Without: an aggregate across all tracked domains (per-tier success, worst failing domains, seen-but-never-fetched count). Returns MCP structured output (structuredContent) for p
Fetch and extract readable markdown from any public URL. GitHub hosts take the GitHub fast path; YouTube video URLs return the transcript and Reddit thread URLs return post+comments (both opt-in via robots, see below); all others use the fetch cascade (Firecrawl → Crawl4AI → raw HTTP). Trimmed to a
Search via SearXNG with local reranking. Fetches a wider result pool, reranks by relevance, returns top N. SearXNG's native direct answers, infoboxes, spelling corrections, and related suggestions are surfaced above the list and in structuredContent.
Search, rerank, then fetch full content of the top result(s) using the fetch cascade (Firecrawl → Crawl4AI → raw HTTP).
Search, fetch top results, then synthesize a summary with citations via Ollama (OLLAMASUMMARIZEMODEL). Falls back to raw fetched content if Ollama is unavailable.