이 MCP로 할 수 있는 일
Analyzes public webpages for robots rules, bot walls, machine-readable content, sources, licenses, and embedded structured data.
기능
도구
10
도구
can_fetch
May this crawler fetch these URLs?
Paid: $0.02 USDC on Base per call, paid automatically by an x402-capable MCP client. No signup, no API key. Answers the RFC 9309 crawl-permission question for up to 100 public http(s) URLs in one payment: may the given crawler agent fetch each URL? One robots.txt fetch per distinct origin (at most 25 origins), not one per URL. Argument agent is the crawler token (e.g. 'GPTBot'); optional, defaults to the wildcard '*' group. An unreadable or unparsable robots.txt is reported as status unknown, allowed null - never guessed as allowed. Errors are never charged for. The free tool robots_check answers the same question for one URL at a time, for nothing. Arguments: urls (1 to 100 strings), agent (optional).
입력 스키마
{'type': 'object', 'title': 'can_fetch_toolArguments', 'required': ['urls'], 'properties': {'urls': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Urls', 'description': '1 to 100 public http(s) URLs to check in one payment. Robots.txt is fetched once per distinct origin, not once per URL (at most 25 distinct origins per call). An unreadable or unparsable robots.txt is reported as unknown, never as allowed.'}, 'agent': {'type': 'string', 'title': 'Agent', 'default': '*', 'description': "Crawler user-agent token to evaluate against robots.txt, e.g. 'GPTBot'. Case-insensitive, stripped and truncated to 64 characters. Optional: omit it, or pass an empty string, to evaluate the wildcard ('*') group."}}}
crawl_policy
Which crawlers may read a fixed sample of the web?
Free, no signup: shares posture_preview's 60-calls-per-hour budget across all callers. Returns live RFC 9309 allow/disallow decisions for a fixed, versioned sample of up to 24 public origins, computed from each origin's own robots.txt. A robots.txt that cannot be read is reported as unknown, never as allowed. Argument: limit (optional, 1-24, default all origins in the seed). The HTTP equivalent is GET /v1/agents/policy (add ?format=csv for CSV). For a single page's full access-posture report, use the paid tool page_posture ($0.01).
입력 스키마
{'type': 'object', 'title': 'crawl_policy_toolArguments', 'properties': {'limit': {'anyOf': [{'type': 'integer', 'maximum': 24, 'minimum': 1}, {'type': 'null'}], 'title': 'Limit', 'default': None, 'description': 'How many origins of the fixed seed to include, 1-24 (default: all of them). The seed list is versioned and published with the dataset.'}}}
domain_report
Domain-level agent-access report
Paid: $0.06 USDC on Base per call. No signup, no API key. The domain-level agent-access report for the ORIGIN of one public http(s) URL: robots.txt decisions for every well-known agent token at the given path, which agent-facing documents the origin publishes (llms.txt, /.well-known/x402, /.well-known/mcp.json, security.txt), whether its declared sitemap actually contains pages the origin allows an agent to fetch, and a sample of those pages analysed for agreement with the robots policy. Verdicts: open, restricted, blocked, mixed, unreachable. Answers a different question than page_posture: not 'can I fetch this one page' but 'what is this whole site's policy toward agents'. Free companion for a single agent at a single path: robots_check. An unanalysable target is never charged for (target_not_analysable); an origin that publishes nothing at all (no robots.txt, no discovery document, no sitemap, unreachable root page) answers nothing_to_report, also uncharged.
입력 스키마
{'type': 'object', 'title': 'domain_report_toolArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
page_data
The data one page already publishes
Paid: $0.03 USDC on Base per call. No signup, no API key. Returns the machine-readable data one public http(s) URL already publishes: every JSON-LD block, OpenGraph/Twitter-card/standard meta, embedded JSON state blobs (e.g. __NEXT_DATA__), parsed entries if the document is RSS/Atom/JSON Feed, HTML tables, and alternate/canonical links - each capped and returned exactly as observed, nothing invented or inferred by a model. Does not render JavaScript, does not follow links off the page, does not summarize. An unanalysable URL is never charged for (error target_not_analysable); a page with nothing extractable answers nothing_to_report, also uncharged.
입력 스키마
{'type': 'object', 'title': 'page_dataArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
page_posture
Access posture for one URL
Paid: $0.01 USDC on Base per call, paid automatically by an x402-capable MCP client. No signup, no API key. Analyses one public http(s) URL and reports whether an agent may treat it as a machine-readable data source: exact-path robots.txt verdict, bot-wall vendor, JavaScript requirement, extractable content, JSON-LD types, licence hint, evidence lines. Verdicts: open, machine_readable, robots_disallowed, bot_walled, auth_required, js_rendered, thin_content, unreachable. An unanalysable URL is never charged for (error target_not_analysable). Argument: url. Free alternative for the verdict alone: posture_preview.
입력 스키마
{'type': 'object', 'title': 'page_postureArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
page_report
Whole-page report for one URL, one payment
Paid: $0.04 USDC on Base per call. No signup, no API key. The whole-page report for one public http(s) URL in ONE payment: the access-posture report (page_posture), where the data is (page_sources) and the data itself (page_data), composed from the same builders as the three single routes, so $0.04 instead of $0.06 for three separate calls. If the posture component cannot be produced the whole call fails closed (target_not_analysable) and nothing settles; if sources or data fails or times out the answer is still 200 with that component absent from `products` and named in `partial`.
입력 스키마
{'type': 'object', 'title': 'page_report_toolArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
page_sources
Where the data is for one URL
Paid: $0.02 USDC on Base per call, paid automatically by an x402-capable MCP client. No signup, no API key. For one public http(s) URL returns the posture verdict plus where the data is: sitemaps (robots.txt Sitemap lines and /sitemap.xml, with the status observed), RSS/Atom feeds, whether /llms.txt answers 200, the status of /.well-known/x402, ai-plugin.json and mcp.json, JSON/CSV/XML endpoints, JSON-LD types, licence hint. Every field is observed or explicitly empty with a reason. An unanalysable URL (target_not_analysable) or a site that advertises nothing (nothing_to_report) is never charged for. Argument: url.
입력 스키마
{'type': 'object', 'title': 'page_sourcesArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
pages_posture
Access posture for up to ten URLs, one payment
Paid: $0.05 USDC on Base per call, as GET /v1/pages/posture?url=...&url=... (repeat url up to ten times) or POST /v1/pages/posture with a JSON body. No signup, no API key. Analyses up to 10 public http(s) URLs in ONE payment and ONE settlement, instead of paying page_posture ten times: the same access-posture report as page_posture, per URL, as a verdict array. An individual unanalysable or empty URL is never a batch failure - it is reported as its own error entry in results, target_not_analysable or nothing_to_report - and the batch is only charged (settles) when at least one URL could be analysed; if every URL fails the whole call answers 404 and nothing settles. Free companions cover a single URL: posture_preview and robots_check.
입력 스키마
{'type': 'object', 'title': 'pages_posture_toolArguments', 'required': ['urls'], 'properties': {'urls': {'type': 'array', 'items': {'type': 'string'}, 'title': 'Urls', 'description': '1 to 10 public http(s) URLs to analyse in one payment. Each is analysed independently: an unreachable or private URL never fails the batch, it is reported as its own error entry in results. Private, loopback and link-local addresses are refused per URL.'}}}
posture_preview
Free preview of the access posture
Free, no payment and no signup: 60 calls per hour, shared across all callers. Returns the headline of the same analysis - verdict, confidence, the full report's field schema, one evidence line and the price - so a caller can decide before paying. Use it when you only need the verdict; call page_posture ($0.01) for the complete report with evidence and provenance.
입력 스키마
{'type': 'object', 'title': 'posture_previewArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}}}
robots_check
May this crawler fetch this URL?
Free, no signup: shares posture_preview's 60-calls-per-hour budget across all callers. Answers the RFC 9309 question for one public http(s) URL and one crawler token (argument agent, e.g. 'GPTBot'; optional, defaults to the wildcard '*' group): may this agent fetch this URL? Also reports the same decision for '*', gptbot, claudebot, googlebot, perplexitybot and ccbot in one call. An unreachable or unparsable robots.txt is reported as status unknown with allowed null - never guessed as allowed. The HTTP equivalent is GET /v1/robots/check. For the wider questions this tool does not answer: page_posture ($0.01) is the full access-posture report, page_sources ($0.02) is where the data is, page_data ($0.03) is the data itself. For many URLs at once, can_fetch ($0.02) answers the same question for up to 100 URLs in one payment, one robots.txt fetch per distinct origin.
입력 스키마
{'type': 'object', 'title': 'robots_check_toolArguments', 'required': ['url'], 'properties': {'url': {'type': 'string', 'title': 'Url', 'description': 'Public http(s) URL to analyse. Must be reachable from the internet; private, loopback and link-local addresses are refused (target_not_analysable) and are never charged for.'}, 'agent': {'type': 'string', 'title': 'Agent', 'default': '*', 'description': "Crawler user-agent token to evaluate against robots.txt, e.g. 'GPTBot'. Case-insensitive, stripped and truncated to 64 characters. Optional: omit it, or pass an empty string, to evaluate the wildcard ('*') group."}}}
기록
최근 도구 변경
추가됨
crawl_policy
2026년 10월 1일 2:43 AM
변경됨
robots_check
2026년 10월 1일 2:43 AM
추가됨
can_fetch
2026년 10월 1일 2:43 AM
추가됨
domain_report
2026년 10월 1일 2:43 AM
추가됨
robots_check
2026년 9월 29일 2:50 AM
추가됨
pages_posture
2026년 9월 29일 2:50 AM
추가됨
page_report
2026년 9월 29일 2:50 AM
추가됨
page_data
2026년 9월 29일 2:50 AM
변경됨
posture_preview
2026년 9월 27일 2:42 AM
추가됨
page_sources
2026년 9월 27일 2:42 AM
변경됨
page_posture
2026년 9월 27일 2:42 AM
추가됨
posture_preview
2026년 9월 23일 2:40 AM
추가됨
page_posture
2026년 9월 23일 2:40 AM
탐색
유사한 MCP 서버
apis-io
io.apis/apis-io
Provides catalog search, comparison, scoring, enrichment, lists, and dataset exports for APIs, providers, specifications, workflo…
Tools for Agents
io.github.max124vip-dev/tools-for-agents
Provides web research and extraction workflows, crawling, browser automation, PDF and book OCR, text processing, embeddings, and …
canton-ccpedia
io.github.UnityNodes/canton-ccpedia
Searches and explains Canton Network and Daml documentation, governance proposals, forums, code examples, APIs, releases, and eco…
webbersites-x402-data-api
io.github.webberdesign/webbersites-x402-data-api
Provides pay-per-call utilities for web scraping, SEO and accessibility audits, document extraction, DNS and email checks, crypto…
api-evangelist
com.apievangelist/api-evangelist
Provides API governance analysis, OpenAPI overlays and version diffs, security-sensitive field classification, code samples, matu…
arch-tools-mcp
io.github.Deesmo/arch-tools-mcp
Provides general-purpose agent APIs for browser automation, web and PDF extraction, data conversion, email operations, fact check…
Agent Margin Router
io.github.AgentMarginRouter/agent-margin-router
Provides pay-per-call tools for blockchain balances, contracts, prices, gas, domains, web extraction, package intelligence, news,…
1cent Web Intelligence for AI Agents
ru.maxzoa/1cent
Performs SSRF-safe public web inspection, URL extraction, metadata and content analysis, accessibility checks, feed and sitemap d…