robots_txt_inspect

MCP tool from AynOps by AynOps

Fetch and parse robots.txt to reveal hidden directories and sitemaps

How to use it

robots_txt_inspect is exposed by the AynOps MCP server. Add the server to your MCP client (Claude Desktop, Cursor, Windsurf and others), and the robots_txt_inspect tool becomes available to the model automatically. See the full listing for setup details and every tool this server provides.

Install AynOps

$docker run --rm -i aynops

FULL AYNOPS LISTING

Other tools in AynOps (15)

asn_lookup

Autonomous System Number (ASN) and network ownership lookup via Team Cymru WHOIS — identifies ASN, BGP prefix, organization, registry, country, and allocation date for domains or IP addresses (no API key required)

cert_transparency

Query crt.sh Certificate Transparency logs for a domain and extract certificate, subdomain, and wildcard information

cloud_exposure_check

Checks for publicly accessible AWS S3, Azure Blob Storage, and Google Cloud Storage buckets using common bucket naming patterns derived from the target domain

cve_lookup

Search NVD for known CVEs by software name and version (no API key required)

dns_enumeration

A, AAAA, MX, NS, TXT, CNAME, SOA, CAA, and common SRV records (SIP, LDAP, XMPP, Kerberos, Autodiscover) + common subdomain brute-forcing

email_security_check

Checks SPF, DKIM, and DMARC DNS records — returns a securityscore, rating, and actionable recommendations for missing or weak configurations

full_recon

Runs all core tools in parallel and returns combined result

headers_analyzer

Analyzes HTTP security headers — checks HSTS, CSP, X-Frame-Options, and more with severity ratings and misconfiguration details

hibp_check

Check if an email or domain appears in Have I Been Pwned breaches (HIBPAPIKEY required)

ip_reputation

Check if an IP is flagged as malicious via AbuseIPDB (api key requied)

port_scan

Nmap-powered scanner with service/version detection and security warnings

ssl_inspect

SSL/TLS certificate — issuer, expiry, cipher strength, SANs, TLS version

tech_stack_detect

Web server, CMS, JS frameworks, CDN, and analytics

trace_redirects

Traces the full HTTP redirect chain hop by hop — flags TLS downgrades, private-IP leaks, redirect loops, cross-domain hops, and overly long chains

whois_lookup

Domain registration data — owner, registrar, creation date, expiry, name servers