MCP Server
Crawler Log Triage
dev.workers.pathwren.www/crawler-log-triage
What this MCP does
Parses crawler access logs to identify operators, purposes, impersonation attempts, request volumes, and generated robots or WAF rules.
Capabilities
Tools
9
tools
example
Run this server's worked example (no arguments)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and you are done: free, read-only, no API key, no account, no consent prompt, nothing for you to invent, and what comes back is this server's own worked example — triage_log, run for real on a canned input from this host's own data — so you see the exact shape of an answer before you send anything of yours. Takes no arguments. Safe to call. Deterministic. Touches no third party. Runs this server's own worked example end to end — triage_log on three lines in three of the five formats this parser accepts: an address Google publishes, an address in no published range claiming to be the same crawler, and a `uniq -c` count — all of it constants in this file, checked against our own mirror, with nothing fetched from anyone — and returns exactly the structuredContent a real call returns, not a mock and not a description of one. Use it to see the shape of an answer before you decide what to send. The input is canned from this host's own data; no URL of yours is fetched and no third party is touched. Example: arguments={} runs triage_log with {"log":"66.249.66.1 Googlebot/2.1\n203.0.113.9 Googlebot/2.1\n412 GPTBot/1.2"} and returns its real answer.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['ran', 'input_came_from', 'what_it_shows', 'answer', 'reproduce', 'this_is_not_a_mock', 'answered_by', 'license'], 'properties': {'ran': {'type': 'object', 'description': 'The tool name and the exact arguments that were run.'}, 'answer': {'type': 'object', 'description': 'The real structuredContent of that call, not a mock.'}, 'license': {'type': 'string'}, 'reproduce': {'type': 'string', 'description': 'A command that reproduces this answer.'}, 'answered_by': {'type': 'object'}, 'what_it_shows': {'type': 'string'}, 'input_came_from': {'type': 'string', 'description': "Where the canned input came from — always this host's own data."}, 'this_is_not_a_mock': {'type': 'string'}}, 'description': "This server's own worked example, executed for real on a canned input from this host's own data.", 'additionalProperties': True}
find_impersonators
Find the lines that are lying
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Only the lines claiming a crawler whose operator publishes address ranges, from an address in none of them — 1997 IPv4 and 1062 IPv6 prefixes, 15 sources. Reverse-DNS operators come back with the command to run: this server makes no outbound request. Example: log='203.0.113.9 Googlebot/2.1' returns one impersonation.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'log': '203.0.113.9 Googlebot/2.1'}], 'required': ['log'], 'properties': {'log': {'type': 'string', 'description': 'The log text. Lines need an address to be checkable.'}}, 'additionalProperties': False}
no_arguments_triage_this_hosts_own_crawler_log
No arguments: triage this host's own published crawler log
TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} to https://www.pathwren.workers.dev/mcp/triage — the answer is the triage of THIS host's own published request log — every operator in it run through the same parser, the same crawler index and the same operator-prefix verification that triage_log applies to a file you paste, rolled up by operator, by category and by crawler, with the share no index entry matches at all and the browser-shaped strings named separately. There is nothing to fill in: the input schema is literally empty, `arguments: {}` and no `arguments` key at all both work, and the subject is a file this host already publishes, so the answer does not depend on you at all. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Nothing is fetched to build it — no request leaves this edge, and none is made to you. The other zero-argument call on this server is triage_my_request, same empty arguments, which answers your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. whoami and example are here too and take nothing either. Every other tool on this server wants a file pasted in; this one wants nothing. The siblings answer one question each under the tool named beside them: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report). Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} returns the rollup by operator, category and crawler over this host's own published window, the cost of blocking each crawler identified, the requests that match no entry in the index, and the robots.txt those findings would generate.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['takes_no_arguments', 'what_this_is', 'the_log_triaged', 'rollup', 'what_no_index_entry_matched', 'no_address_was_triaged', 'answered_by', 'this_call_touched', 'reproduce', 'caveats', 'license', 'independent'], 'properties': {'rollup': {'type': 'object', 'description': 'Requests by operator, by category and by crawler, with the identified share.'}, 'caveats': {'type': 'array', 'items': {'type': 'string'}}, 'license': {'type': 'string'}, 'reproduce': {'type': 'string'}, 'answered_by': {'type': 'object'}, 'independent': {'type': 'boolean'}, 'what_this_is': {'type': 'string'}, 'the_log_triaged': {'type': 'object', 'description': 'Where the log came from, how it was parsed, and what it covers.'}, 'this_call_touched': {'type': 'object'}, 'takes_no_arguments': {'type': 'boolean'}, 'no_address_was_triaged': {'type': 'string'}, 'what_no_index_entry_matched': {'type': 'object'}, 'to_do_this_for_your_own_file': {'type': 'string'}}, 'description': "This host's own published request log run through the same triage engine a pasted log gets, rolled up by operator, category and crawler.", 'additionalProperties': True}
robots_from_log
robots.txt from a log
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — A robots.txt naming only the crawlers in your log, each with its request count and cost of blocking, plus a warning for any that do not documentably obey it — there the file is a request, not enforcement. Example: log='412 GPTBot/1.2', stance='block-ai-training' blocks GPTBot only.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'log': '412 GPTBot/1.2', 'stance': 'block-ai-training'}], 'required': ['log'], 'properties': {'log': {'type': 'string', 'description': 'The log text.'}, 'stance': {'enum': ['block-ai-training', 'block-all-ai', 'block-unverifiable', 'allow-all'], 'type': 'string', 'description': 'Default block-ai-training.'}}, 'additionalProperties': False}
summarize_by_operator
Roll a log up by operator and category
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Aggregate only: who crawled you, how many requests each, what share, which category, and what blocking each would cost. Eats a `uniq -c` table straight from a shell pipeline. Example: log='412 GPTBot/1.2' returns OpenAI, 412 requests, 100%, ai-training.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'log': '412 GPTBot/1.2'}], 'required': ['log'], 'properties': {'log': {'type': 'string', 'description': 'Log text, or a `uniq -c` user-agent table.'}}, 'additionalProperties': False}
triage_log
Triage a paste of log lines
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — Paste log lines — combined, JSON, `uniq -c`, `<ip> <ua>` or bare user-agents, auto-detected — for per-line crawler, operator, purpose and cost of blocking, a rollup by operator and category, and every line whose address contradicts its claim. Log text, never a URL. Example: log='66.249.66.1 Googlebot/2.1' returns Googlebot, Google, search, verified.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'log': '66.249.66.1 Googlebot/2.1\n203.0.113.9 Googlebot/2.1\n412 GPTBot/1.2'}], 'required': ['log'], 'properties': {'log': {'type': 'string', 'description': 'The log text, up to 5000 lines. Mixed formats are fine.'}, 'limit': {'type': 'integer', 'maximum': 500, 'minimum': 1, 'description': 'Max rows, default 200.'}, 'detail': {'enum': ['rows', 'summary'], 'type': 'string', 'description': 'Per-line table, or aggregates only.'}}, 'additionalProperties': False}
triage_my_request
Triage my own request (no arguments)
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp/triage and the answer is your own request triaged as one line of an access log — the crawler this host's index identifies from your user-agent, its operator and category, and whether the address you came from verifies against that operator's published prefixes. THIS IS THE CALL TO MAKE FIRST on this server: it is the only one whose subject is the request you just sent, so there is nothing to look up and no argument to invent. No key, no account, no OAuth, no session to open first, read-only, and nothing for you to invent. Every fact is computed from your own headers and from files this host already publishes; nothing is fetched to build it, and no URL of yours is touched. Only this server answers your own request triaged as one line of an access log, by the same parser and the same index that read a whole file — its six siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev; whoami is answered by all seven and is never an error, so one identical body still walks the whole family if you prefer that. Example: the complete call, exactly as written, nothing to fill in — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"triage_my_request","arguments":{}}} returns the single log line built from your address and your user-agent, the triage row for it, and the same rollup triage_log returns for a whole file.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['you', 'we_book_you_as', 'answered_by', 'this_call_touched', 'caveats', 'license', 'independent'], 'properties': {'you': {'type': 'object', 'description': 'The user-agent you sent and the address you came from.'}, 'caveats': {'type': 'array', 'items': {'type': 'string'}}, 'license': {'type': 'string'}, 'answered_by': {'type': 'object', 'description': 'Which server answered, at which endpoint, with which tool.'}, 'independent': {'type': 'boolean'}, 'we_book_you_as': {'type': 'object', 'description': "The class this host's own instrument records for that user-agent."}, 'this_call_touched': {'type': 'object', 'description': 'Exactly which files were read. No third party is contacted.'}}, 'description': "One server's own question, answered about the caller, from the headers of this request and from files this host already publishes.", 'additionalProperties': True}
waf_ruleset_from_log
WAF ruleset from a log
FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_triage_this_hosts_own_crawler_log","arguments":{}}} — nginx, Caddy, Cloudflare, HAProxy or Apache rules for only the crawlers in your log. The reply warns that a UA rule stops only an honest client, and that impersonation is an address problem needing the published prefixes as an allowlist. Example: log='412 GPTBot/1.2', target='nginx', scope='ai-training'.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{'log': '412 GPTBot/1.2', 'scope': 'ai-training', 'target': 'nginx'}], 'required': ['log'], 'properties': {'log': {'type': 'string', 'description': 'The log text.'}, 'scope': {'type': 'string', 'description': "A category (default ai-training), 'all-seen', 'impersonators', or a stance."}, 'action': {'enum': ['block', 'challenge'], 'type': 'string', 'description': 'Default block.'}, 'target': {'enum': ['nginx', 'caddy', 'cloudflare', 'haproxy', 'apache'], 'type': 'string', 'description': 'Default nginx.'}}, 'additionalProperties': False}
whoami
Who is calling? — the one call that needs nothing from you
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp/triage and the answer is the user-agent you sent, the class this host's own instrument books it as, whether we have logged you here before — and the part only this server can answer: the request you just made, run through triage_log as a single log line, naming the crawler it identifies, its operator and category, and whether your address verifies against that operator's published prefixes. Takes no arguments. Safe to call. Deterministic. Touches no third party. Nothing is fetched to build it: every fact comes from the headers on your own request or from a file this host already publishes. Only this server answers the last part — its five siblings answer one other question each about you, same empty arguments, each under the tool named beside its path: /mcp (whoami), /mcp/doctor (no_arguments_check_this_hosts_own_discovery_documents), /mcp/lint (whoami), /mcp/robots (no_arguments_lint_this_hosts_robots_txt), /mcp/netcheck (no_arguments_report_the_crawler_ip_ranges_this_host_mirrors), /mcp/markdown (markdown_lane_self_report) on https://www.pathwren.workers.dev. Example: arguments={} returns your user-agent, your address, the class we book you as, whether we have seen you here before, and your own request triaged as one line of an access log, by the same parser and the same index that read a whole file.
Read only
Idempotent
Input schema
{'type': 'object', 'examples': [{}], 'required': [], 'properties': {}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['you', 'we_book_you_as', 'we_have_seen_you', 'answered_by', 'this_call_touched', 'caveats', 'license', 'independent'], 'properties': {'you': {'type': 'object', 'description': 'The user-agent you sent and the address you came from.'}, 'caveats': {'type': 'array', 'items': {'type': 'string'}, 'description': 'What this answer does NOT establish — a user-agent is a claim.'}, 'license': {'type': 'string'}, 'answered_by': {'type': 'object', 'description': 'Which server answered, at which endpoint.'}, 'independent': {'type': 'boolean', 'description': 'This host is independent and unaffiliated.'}, 'we_book_you_as': {'type': 'object', 'description': "The class this host's own instrument records for that user-agent."}, 'we_have_seen_you': {'type': 'object', 'description': 'Whether this user-agent appears in the published observation window.'}, 'this_call_touched': {'type': 'object', 'description': 'Exactly which files were read to answer. No third party is contacted.'}}, 'description': 'Facts about the caller, derived only from the headers of this request and from files this host already publishes.', 'additionalProperties': True}
History
Recent tool changes
Added
waf_ruleset_from_log
Sept. 17, 2026, 12:39 p.m.
Added
robots_from_log
Sept. 17, 2026, 12:39 p.m.
Added
summarize_by_operator
Sept. 17, 2026, 12:39 p.m.
Added
find_impersonators
Sept. 17, 2026, 12:39 p.m.
Added
triage_log
Sept. 17, 2026, 12:39 p.m.
Added
example
Sept. 17, 2026, 12:39 p.m.
Added
whoami
Sept. 17, 2026, 12:39 p.m.
Added
triage_my_request
Sept. 17, 2026, 12:39 p.m.
Added
no_arguments_triage_this_hosts_own_crawler_log
Sept. 17, 2026, 12:39 p.m.
Discover
Similar MCP servers
Crypto Bot Audit + Market Data (x402 paid)
io.github.kaminariouji/x402-audit-agent
Audits crypto bot source code and provides paid crypto market, token, gas, stablecoin, and DeFi TVL data.
TunnelMind Data API
ai.tunnelmind/data
Aggregates web, routing, supply-chain, tracker, threat, and agent-registry intelligence into risk verdicts, evidence, receipts, a…
Polyform
org.polyform/polyform
Offers pay-per-call APIs for economic, financial, geographic, healthcare, domain, email, sanctions, blockchain, and business risk…
HubVibe: Pay-per-Call Tools for AI Agents: Web Search, Email Verify, KYC, Stocks, Crypto, News, Data
io.github.Its-fortunatefolly/hubvibe
Offers paid utilities for web audits, HTTP fetching and extraction, BigQuery analysis, LLM processing, code execution, blockchain…
Qiniso
io.github.qinisolabs/qiniso
Provides deterministic formatting, parsing, holiday and tax lookups, address handling, and checksum or structure validation for i…
SYNTHORA x402 Intelligence Mesh
com.hergertsynthora/synthora-x402
Delivers paid intelligence on markets, crypto, prediction markets, maritime and space risks, domain and wallet exposure, sanction…
Satoshidata Wallet Intel
io.github.wrbtc/wallet-intelligence
Provides Bitcoin wallet intelligence, address labels, trust and risk signals, transaction verification, entity activity, fees, me…
Cloudflare Radar
io.github.pipeworx-io/cloudflare-radar
Provides Cloudflare Radar internet observatory data on DDoS attacks, BGP leaks, domain popularity, internet quality, and traffic …