MCP Server
Crawler IP Verifier - real Googlebot or fake
dev.workers.pathwren.www/crawler-ip-verifier
What this MCP does
Verifies crawler IP addresses against published operator ranges, analyzes CIDR overlaps, reports verification methods, and exports allowlists or denylists.
Capabilities
Tools
9
tools
example
Run this server's worked example (no arguments)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — verify_batch, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — verify_batch on one address from inside a prefix Google itself publishes (mirrored here every six hours) and one RFC 5737 documentation address that is in no published range — our own mirror, no third party asked — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs verify_batch with {"addresses":[{"ip":"66.249.66.1","claim":"Googlebot"},{"ip":"203.0.113.9","claim":"GPTBot and returns its real answer.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['ran', 'input_came_from', 'what_it_shows', 'answer', 'reproduce', 'this_is_not_a_mock', 'answered_by', 'license'], 'properties': {'ran': {'type': 'object', 'description': 'The tool name and the exact arguments that were run.'}, 'answer': {'type': 'object', 'description': 'The real structuredContent of that call, not a mock.'}, 'license': {'type': 'string'}, 'reproduce': {'type': 'string', 'description': 'A command that reproduces this answer.'}, 'answered_by': {'type': 'object'}, 'what_it_shows': {'type': 'string'}, 'input_came_from': {'type': 'string', 'description': "Where the canned input came from — always this host's own data."}, 'this_is_not_a_mock': {'type': 'string'}}, 'description': "This server's own worked example, executed for real on a canned input from this host's own data.", 'additionalProperties': True}
export_ip_acl
Paste-ready allowlist or denylist
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Turn a set of operators into a config you can paste: nginx geo, nginx allow/deny, Apache, HAProxy, a Cloudflare firewall expression, an ipset script, a Caddy matcher, or a plain CIDR list. Every export carries a provenance header naming each source URL and the mirror time, and reports the rule cost. Example: operators='all', format='cidr-list', action='allow'.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'action': 'allow', 'format': 'cidr-list', 'operators': 'all'}], 'properties': {'action': {'enum': ['allow', 'deny'], 'type': 'string', 'description': 'Defaults to allow.'}, 'format': {'enum': ['cidr-list', 'nginx-geo', 'nginx-allow-deny', 'apache', 'haproxy', 'cloudflare-expression', 'ipset', 'caddy'], 'type': 'string', 'description': 'Output format. Defaults to cidr-list.'}, 'operators': {'anyOf': [{'type': 'string'}, {'type': 'array', 'items': {'type': 'string'}}], 'description': 'Source slugs, or "all". Defaults to every mirrored source.'}, 'ip_version': {'enum': ['both', 'ipv4', 'ipv6'], 'type': 'string', 'description': 'Defaults to both.'}, 'variable_name': {'type': 'string', 'description': 'Variable/set name for nginx geo and ipset. Defaults to ai_crawler.'}}}
lookup_prefix
Prefix arithmetic against published ranges
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Give a CIDR and get every published operator prefix that contains it, is contained by it, or partially overlaps it — the check for 'does my network collide with a crawler range' and for auditing an allowlist somebody handed you. Give an operator or source slug instead and get everything that source publishes. Example: cidr='66.249.66.0/24'.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'cidr': '66.249.66.0/24'}], 'properties': {'cidr': {'type': 'string', 'description': 'A CIDR or a bare address, e.g. 20.171.206.0/24 or 2600:1f00::/32.'}, 'operator': {'type': 'string', 'description': 'A source slug, e.g. openai-gptbot, google-googlebot.'}}}
no_arguments_report_the_crawler_ip_ranges_this_host_mirrors
No arguments: every operator prefix list this host mirrors, and how stale each one is
TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} to https://www.pathwren.workers.dev/mcp/netcheck — the answer is the state of every crawler-operator prefix list this host mirrors — how many IPv4 and IPv6 prefixes each source publishes and how many addresses that is, when each was last fetched and how many minutes ago that was, which crawlers each source covers, which prefixes more than one operator claims, and the verification method each operator documents for the crawlers that publish no list at all. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. The other zero-argument call on this server is verify_my_address, same empty arguments, which answers the address you are calling from, checked against every operator-published prefix list this host mirrors with your user-agent taken as the claim — the verdict, the prefix that matched, which operator published it and when it was last mirrored. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} returns a row per source with its prefix counts, address total, fetch time and staleness in minutes, the prefixes published by two operators at once, and the split between operators you can verify by prefix and operators you can only verify by reverse DNS.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['takes_no_arguments', 'what_this_is', 'the_mirror', 'per_source', 'how_each_operator_can_be_verified', 'what_a_miss_means', 'answered_by', 'this_call_touched', 'reproduce', 'caveats', 'license', 'independent'], 'properties': {'caveats': {'type': 'array', 'items': {'type': 'string'}}, 'license': {'type': 'string'}, 'reproduce': {'type': 'string'}, 'per_source': {'type': 'array', 'description': 'One row per operator list: counts, fetch time, staleness, crawlers covered.'}, 'the_mirror': {'type': 'object', 'description': 'Totals across every source, with staleness in minutes.'}, 'answered_by': {'type': 'object'}, 'independent': {'type': 'boolean'}, 'what_this_is': {'type': 'string'}, 'this_call_touched': {'type': 'object'}, 'what_a_miss_means': {'type': 'string'}, 'takes_no_arguments': {'type': 'boolean'}, 'how_each_operator_can_be_verified': {'type': 'object'}, 'to_do_this_for_your_own_addresses': {'type': 'string'}, 'prefixes_published_by_more_than_one_source': {'type': 'array'}}, 'description': 'The state of every crawler-operator prefix list this host mirrors, with per-source freshness and the verification method each operator documents.', 'additionalProperties': True}
range_stats
The state of the mirror
Takes no arguments. Safe to call. Deterministic. Touches no third party. Per source: prefix counts, total addresses and /24-equivalents, smallest and largest prefix, the publishing URL and when it was last fetched; plus prefixes published by more than one operator. This is the sizing and freshness information an allowlist decision needs. Example: {} — no arguments at all.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
verification_methods
Can this crawler be verified at all, and how?
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Per crawler: whether the operator publishes IP ranges, documents reverse DNS, or publishes no verification method at all — with the exact source URL or the exact dig commands, and how fresh this mirror is. Call it before reading anything into an address that matched nothing. Reverse DNS is never run here; the command is handed back instead. Example: crawler='claudebot'. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verification-methods?crawler=claudebot&s=client-dossiers
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'crawler': 'claudebot'}], 'properties': {'crawler': {'type': 'string', 'description': 'Crawler slug, name, operator or UA substring. Omit for all of them.'}}}
verify_batch
Verify many addresses at once
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_report_the_crawler_ip_ranges_this_host_mirrors","arguments":{}}} — Check up to 500 addresses against every mirrored operator prefix list in one call. Returns, per address, the matching prefix, the source that published it and when it was mirrored, plus a rollup by operator. Accepts {ip, claim} pairs to test a claimed identity against the ranges, and says plainly that a miss is not evidence of a fake. Example: addresses=[{ip:'66.249.66.1', claim:'Googlebot'}, {ip:'203.0.113.9', claim:'GPTBot'}] — the first matches a published Google prefix, the second matches nothing. Also callable without MCP, same implementation: GET https://www.pathwren.workers.dev/tools/verify-crawler?ip=66.249.66.1&ua=Googlebot&s=client-dossiers
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'addresses': [{'ip': '66.249.66.1', 'claim': 'Googlebot'}, {'ip': '203.0.113.9', 'claim': 'GPTBot'}]}], 'required': ['addresses'], 'properties': {'addresses': {'anyOf': [{'type': 'string'}, {'type': 'array', 'items': {'anyOf': [{'type': 'string'}, {'type': 'object'}]}}], 'description': 'IPv4/IPv6 addresses: an array, a whitespace or comma separated string, or objects like {"ip":"20.171.206.1","claim":"GPTBot"}. Max 500.'}}}
verify_my_address
Does my address verify? (no arguments)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_my_address","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and the answer is the address you are calling from, checked against every operator-published prefix list this host mirrors with your user-agent taken as the claim — the verdict, the prefix that matched, which operator published it and when it was last mirrored. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers whether the address you are calling from falls inside a prefix a crawler operator publishes, and which prefix from which source — its six siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/triage (no_arguments_triage_this_hosts_own_crawler_log), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev; whoami is answered by all seven and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"verify_my_address","arguments":{}}} returns a verdict for the address you are calling from, the matching prefix with its source and mirror time, and the sentence that a miss is not evidence of a fake.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['you', 'we_book_you_as', 'answered_by', 'this_call_touched', 'caveats', 'license', 'independent'], 'properties': {'you': {'type': 'object', 'description': 'The user-agent you sent and the address you came from.'}, 'caveats': {'type': 'array', 'items': {'type': 'string'}}, 'license': {'type': 'string'}, 'answered_by': {'type': 'object', 'description': 'Which server answered, at which endpoint, with which tool.'}, 'independent': {'type': 'boolean'}, 'we_book_you_as': {'type': 'object', 'description': "The class this host's own instrument records for that user-agent."}, 'this_call_touched': {'type': 'object', 'description': 'Exactly which files were read. No third party is contacted.'}}, 'description': "One server's own question, answered about the caller, from the headers of this request and from files this host already publishes.", 'additionalProperties': True}
whoami
Who is calling? (no arguments)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/netcheck and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back describes YOU, the caller — the user-agent you sent, the address you came from, the class our instrument books you as, and whether this host has seen you before. Takes no arguments. Safe to call. Deterministic. Touches no third party. Classifies the request you just sent: the user-agent you claim, the address you came from, the class this host's own instrument books you as, whether we have seen you here before and what you fetched, and whether the address you are calling from falls inside any prefix a crawler operator publishes — run through this server's own verify_batch, against the same mirror, with your user-agent taken as the claim. Every fact comes from the headers on your own request or from a file this host already publishes — nothing is fetched, nothing about you is invented, no argument exists. Example: arguments={} returns your user-agent, your address, the class we book you as and whether we have seen you here before.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['you', 'we_book_you_as', 'we_have_seen_you', 'answered_by', 'this_call_touched', 'caveats', 'license', 'independent'], 'properties': {'you': {'type': 'object', 'description': 'The user-agent you sent and the address you came from.'}, 'caveats': {'type': 'array', 'items': {'type': 'string'}, 'description': 'What this answer does NOT establish — a user-agent is a claim.'}, 'license': {'type': 'string'}, 'answered_by': {'type': 'object', 'description': 'Which server answered, at which endpoint.'}, 'independent': {'type': 'boolean', 'description': 'This host is independent and unaffiliated.'}, 'we_book_you_as': {'type': 'object', 'description': "The class this host's own instrument records for that user-agent."}, 'we_have_seen_you': {'type': 'object', 'description': 'Whether this user-agent appears in the published observation window.'}, 'this_call_touched': {'type': 'object', 'description': 'Exactly which files were read to answer. No third party is contacted.'}}, 'description': 'Facts about the caller, derived only from the headers of this request and from files this host already publishes.', 'additionalProperties': True}
History
Recent tool changes
Added
range_stats
Sept. 17, 2026, 12:39 p.m.
Added
verification_methods
Sept. 17, 2026, 12:39 p.m.
Added
export_ip_acl
Sept. 17, 2026, 12:39 p.m.
Added
lookup_prefix
Sept. 17, 2026, 12:39 p.m.
Added
verify_batch
Sept. 17, 2026, 12:39 p.m.
Added
example
Sept. 17, 2026, 12:39 p.m.
Added
whoami
Sept. 17, 2026, 12:39 p.m.
Added
verify_my_address
Sept. 17, 2026, 12:39 p.m.
Added
no_arguments_report_the_crawler_ip_ranges_this_host_mirrors
Sept. 17, 2026, 12:39 p.m.
Discover
Similar MCP servers
hyperion
com.thetempleofdoom.hyperion/hyperion
Acts as a paid MCP tool marketplace and utility gateway with server discovery, HTTP and JavaScript tools, research, data conversi…
Vee3
io.github.Vee3io/vee3
Manages Clerk authentication infrastructure, including users, organizations, domains, sessions, tokens, OAuth, SSO, machines, per…
IA-QA — 130+ QA & Dev Tools for AI Agents
io.github.JcJamet/ia-qa-toolbox
Provides deterministic QA, evaluation, testing, code analysis, prompt and RAG checks, model comparison, and web security diagnost…
validoria-mcp
com.validoria/validoria-mcp
Runs continuous website, API, and webshop tests covering security, SEO, performance, accessibility, browser journeys, and inciden…
HubVibe: Pay-per-Call Tools for AI Agents: Web Search, Email Verify, KYC, Stocks, Crypto, News, Data
io.github.Its-fortunatefolly/hubvibe
Offers paid utilities for web audits, HTTP fetching and extraction, BigQuery analysis, LLM processing, code execution, blockchain…
developer-tools
net.programmes/developer-tools
Provides general-purpose developer utilities for encoding, hashing, encryption, JSON, HTML, CSS, networking, and related data tra…
Qiniso
io.github.qinisolabs/qiniso
Provides deterministic formatting, parsing, holiday and tax lookups, address handling, and checksum or structure validation for i…
ContrastAPI
com.contrastcyber/api
Provides security research and assessment tools covering CVEs, IOCs, dependencies, secrets, injection risks, HTTP headers, domain…