crawl_url
Call immediately after search when you need the actual article body for quoting, summarizing, or extracting data.
How to use it
crawl_url is exposed by the Web Research Assistant MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the crawl_url tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.
Install Web Research Assistant
docker run -d -p 2288:8080 searxng/searxng:latestOther tools in Web Research Assistant (12)
Auto-discover and crawl official API documentation. Dynamically finds docs URLs using patterns (docs.{api}.com, {api}.com/docs, etc.), searches for specific topics, crawls pages, and extracts overview, parameters, examples, and related links. Works for ANY API - no hardcoded URLs. Perfect for API in
NEW! Instantly check if external services are experiencing issues. Covers 25+ popular services (Stripe, AWS, GitHub, OpenAI, Vercel, etc.). Returns operational status, current incidents, and component health. Critical for production debugging - know immediately if the issue is external. Response tim
Compare 2-5 technologies side-by-side. Auto-detects category (framework/database/language) and gathers data from NPM, GitHub, and web search. Returns structured comparison with popularity metrics (downloads, stars), performance insights, and best-use summaries. Fast parallel processing (3-4s).
Extract structured data from HTML pages. Supports tables, lists, fields (via CSS selectors), JSON-LD, and auto-detection. Returns clean JSON output. More efficient than parsing full page text. Perfect for scraping pricing tables, package specs, release notes, or any structured content.
NEW! Get release notes and changelogs for package upgrades. Fetches GitHub releases, highlights breaking changes, and provides upgrade recommendations. Answers "What changed in version X → Y?" and "Are there breaking changes?" Perfect for planning dependency updates.
Get GitHub repository health metrics including stars, forks, issues, recent commits, and project details. Use when evaluating open source projects.
Look up specific npm, PyPI, crates.io, or Go package metadata including version, downloads, license, and dependencies. Use when you know the package name.
Search for packages by keywords or functionality (e.g., "web framework", "json parser"). Use when you need to find packages that solve a specific problem.
Find code examples, tutorials, and technical articles. Optimized for technical content with optional time filtering. Perfect for learning APIs or finding usage patterns.
Find high-quality royalty-free stock images from Pixabay. Returns photos, illustrations, or vectors. Requires PIXABAYAPIKEY environment variable.
Find Stack Overflow solutions for error messages and stack traces. Auto-detects language/framework, extracts key terms (CORS, map, undefined, etc.), filters irrelevant results, and prioritizes Stack Overflow solutions. Handles web-specific errors (CORS, fetch).
Use first to gather recent information and URLs from SearXNG. Returns 1–10 ranked snippets with clickable URLs.