MCPVault

crawl_url_with_fallback

MCP tool from Crawl MCP by walksoda

Crawl with fallback strategies for anti-bot sites

How to use it

crawl_url_with_fallback is exposed by the Crawl MCP MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the crawl_url_with_fallback tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.

Install Crawl MCP

$uvx --from git+https://github.com/walksoda/crawl-mcp crawl-mcp
FULL CRAWL MCP LISTING

Other tools in Crawl MCP (18)

batch_crawl

Crawl multiple URLs with fallback (max 3 URLs)

batch_extract_youtube_transcripts

Extract transcripts from multiple YouTube videos (max 3)

batch_search_google

Perform multiple Google searches (max 3)

crawl_url

Extract web page content with JavaScript support

deep_crawl_site

Crawl multiple pages from a site with configurable depth

enhanced_process_large_content

Process large content with chunking and BM25 filtering

extract_entities

Extract entities (emails, phones, etc.) from web pages

extract_structured_data

Extract structured data using CSS selectors or LLM

extract_youtube_comments

Extract YouTube video comments with pagination

extract_youtube_transcript

Extract YouTube transcripts with timestamps

get_search_genres

Get available search genres

get_supported_file_formats

Get supported file formats and capabilities

get_youtube_video_info

Get YouTube video metadata and transcript availability

intelligent_extract

Extract specific data from web pages using LLM

multi_url_crawl

Multi-URL crawl with pattern-based config (max 5 URL patterns)

process_file

Convert PDF, Word, Excel, PowerPoint, ZIP to markdown

search_and_crawl

Search Google and crawl top results

search_google

Search Google with genre filtering