crawl_site
Comprehensive site crawling with persistence (always requires outputdir).
How to use it
crawl_site is exposed by the Crawl4ai MCP Server MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the crawl_site tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.
Install Crawl4ai MCP Server
pip install -r requirements.txtOther tools in Crawl4ai MCP Server (19)
Enable adaptive crawling (default: false)
Optional Crawl4AI browser config overrides
Multi-page breadth-first crawling with filtering and adaptive stopping.
Sitemap-based crawling with persistence (always requires outputdir).
Optional Crawl4AI crawler config overrides
(required): Starting URL for site crawl
Regex patterns to exclude URLs
Regex patterns URLs must match
Maximum link depth to follow (default: 1, max: 4)
Maximum sitemap entries to process (default: 1000)
Maximum pages to crawl (default: 5, max: 100)
(optional): If provided, persists content to disk and returns metadata only
Stay within the same domain (default: true)
Fetch a single URL and return markdown content.
Optional C4A-Script for page interaction
(required): Starting URL for the crawl
(required): URL to sitemap.xml
Request timeout (default: 45s, max: 600s)
(required): The URL to scrape