crawl_url_with_fallback

MCP tool from Crawl MCP by walksoda

Crawl with fallback strategies for anti-bot sites

How to use it

Project documentation lists crawl_url_with_fallback for the Crawl MCP MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), then check which tools your installed version makes available. Tool availability can depend on configuration and credentials. A server handshake does not verify this tool’s behavior. See the full listing for setup details.

Install Crawl MCP

$uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcp
FULL CRAWL MCP LISTING

Other tools in Crawl MCP (18)

batch_crawl

Crawl multiple URLs with fallback (max 3 URLs)

batch_extract_youtube_transcripts

Extract transcripts from multiple YouTube videos (max 3)

batch_search_google

Perform multiple Google searches (max 3)

crawl_url

Extract web page content with JavaScript support

deep_crawl_site

Crawl multiple pages from a site with configurable depth

enhanced_process_large_content

Process large content with chunking and BM25 filtering

extract_entities

Extract entities (emails, phones, etc.) from web pages

extract_structured_data

Extract structured data using CSS selectors or LLM

extract_youtube_comments

Extract YouTube video comments with pagination

extract_youtube_transcript

Extract YouTube transcripts with timestamps

get_search_genres

Get available search genres

get_supported_file_formats

Get supported file formats and capabilities

get_youtube_video_info

Get YouTube video metadata and transcript availability

intelligent_extract

Extract specific data from web pages using LLM

multi_url_crawl

Multi-URL crawl with pattern-based config (max 5 URL patterns)

process_file

Convert PDF, Word, Excel, PowerPoint, ZIP to markdown

search_and_crawl

Search Google and crawl top results

search_google

Search Google with genre filtering