MCP Radar
Servidor MCP

Crawl Census

io.github.taylorsmithgg/crawl-census
Herramientas para desarrolladores Búsqueda e investigación Público y accesible MCP 2026-07-28

Qué hace este MCP

Audits websites for AI crawler access using robots.txt, live request checks, readability, structured data, and llms.txt, and provides crawler-access statistics.

Capacidades

Herramientas

7 herramientas
agent_profile
What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.
Esquema de entrada 
{'type': 'object', 'required': ['agent'], 'properties': {'agent': {'type': 'string', 'description': 'Crawler token, e.g. gptbot, claudebot, ccbot, google-extended.'}}}
census_facts
Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.
Esquema de entrada 
{'type': 'object', 'properties': {}}
census_stats
Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
Esquema de entrada 
{'type': 'object', 'properties': {}}
crawl_preflight
Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.
Esquema de entrada 
{'type': 'object', 'required': ['agent', 'domains'], 'properties': {'agent': {'type': 'string', 'description': 'Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.'}, 'domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.'}}}
scan_site
Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.
Esquema de entrada 
{'type': 'object', 'required': ['domain'], 'properties': {'domain': {'type': 'string', 'description': 'Bare hostname, for example example.com'}}}
site_report
Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
Esquema de entrada 
{'type': 'object', 'required': ['domain'], 'properties': {'domain': {'type': 'string', 'description': 'Bare hostname'}}}
submit_domains
Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.
Esquema de entrada 
{'type': 'object', 'required': ['domains'], 'properties': {'domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.'}}}
Historial

Cambios recientes en herramientas

Añadido
submit_domains
17 de September de 2026 a las 12:52
Añadido
census_facts
17 de September de 2026 a las 12:52
Añadido
agent_profile
17 de September de 2026 a las 12:52
Añadido
crawl_preflight
17 de September de 2026 a las 12:52
Añadido
census_stats
17 de September de 2026 a las 12:52
Añadido
site_report
17 de September de 2026 a las 12:52
Añadido
scan_site
17 de September de 2026 a las 12:52
Descubrir

Servidores MCP similares

apis-io

io.apis/apis-io

Provides catalog search, comparison, scoring, enrichment, lists, and dataset exports for APIs, providers, specifications, workflo…

Tools for Agents

io.github.max124vip-dev/tools-for-agents

Provides web research and extraction workflows, crawling, browser automation, PDF and book OCR, text processing, embeddings, and …

canton-ccpedia

io.github.UnityNodes/canton-ccpedia

Searches and explains Canton Network and Daml documentation, governance proposals, forums, code examples, APIs, releases, and eco…

webbersites-x402-data-api

io.github.webberdesign/webbersites-x402-data-api

Provides pay-per-call utilities for web scraping, SEO and accessibility audits, document extraction, DNS and email checks, crypto…

api-evangelist

com.apievangelist/api-evangelist

Provides API governance analysis, OpenAPI overlays and version diffs, security-sensitive field classification, code samples, matu…

arch-tools-mcp

io.github.Deesmo/arch-tools-mcp

Provides general-purpose agent APIs for browser automation, web and PDF extraction, data conversion, email operations, fact check…

Agent Margin Router

io.github.AgentMarginRouter/agent-margin-router

Provides pay-per-call tools for blockchain balances, contracts, prices, gas, domains, web extraction, package intelligence, news,…

1cent Web Intelligence for AI Agents

ru.maxzoa/1cent

Performs SSRF-safe public web inspection, URL extraction, metadata and content analysis, accessibility checks, feed and sitemap d…