create_crawl
Start an async crawl that autonomously discovers and scrapes entire websites by following links. Returns a crawlid — the crawl runs in the background and does not return content in this response. You must then call getcrawlresults with the crawlid to poll status and retrieve the scraped pages (same
How to use it
create_crawl is exposed by the Olostep MCP Server MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the create_crawl tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.
Install Olostep MCP Server
docker run -i --rm -e OLOSTEP_API_KEY="your-api-key" olostep/mcp-server:localOther tools in Olostep MCP Server (13)
, object, task, result (JSON if provided), sources, created
Search the web and return AI-powered answers in the JSON structure you want, with sources and citations.
, status, totalurls, createdat, formats, country, parser, urls
Scrape up to 10k URLs at the same time. Perfect for large-scale data extraction.
, object, status, starturl, maxpages, created, formats, country, parser
Get all URLs on a website. Extract all URLs for discovery and analysis.
Retrieve the results of a previously submitted batch scrape job using its batchid.
Retrieve the status and scraped pages for an async crawl started with createcrawl. This is the required companion to createcrawl — createcrawl only kicks off the job and returns a crawlid; this tool is how you actually fetch the discovered pages and their content.
Retrieves webpage content in clean markdown format with support for JavaScript rendering.
Search and retrieve relevant URLs from a website, sorted by relevance to your query.
, object, url, totalurls, urls, searchquery, topn
Extract content from a single URL. Supports multiple formats and JavaScript rendering.
Search the Web for a given query and get structured results (non-AI, parser-based).