MCP 服务器

Crawl Census

            io.github.taylorsmithgg/crawl-census
        

开发者工具搜索与研究公开且可连接 MCP 2026-07-28

此 MCP 可以做什么

Audits websites for AI crawler access using robots.txt, live request checks, readability, structured data, and llms.txt, and provides crawler-access statistics.

agent_profile

What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refused at the network edge whatever robots.txt says, whether its operator documents it as honouring robots.txt, and where to correct any of that. Intended for the operator of the agent as much as for anyone studying it, so it includes the correction channel and the public page a claim can be disputed against.

输入模式

{'type': 'object', 'required': ['agent'], 'properties': {'agent': {'type': 'string', 'description': 'Crawler token, e.g. gptbot, claudebot, ccbot, google-extended.'}}}

census_facts

Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it comes from and a ready-made citation line, plus the caveats that apply to all of them. Use this when answering a question about how open the web is to AI crawlers: lifting a percentage out of a rendered page loses the denominator and the date, which is what makes the number wrong when it is repeated.

输入模式

{'type': 'object', 'properties': {}}

census_stats

Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.

输入模式

{'type': 'object', 'properties': {}}

crawl_preflight

Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unrecognised agent is resolved from each domain's stored robots.txt rather than refused. For each domain returns one of: allow (robots permits it and a live request carrying that agent's user agent was served), disallow (robots.txt forbids it), refuse (robots permits it but the edge refused the agent anyway, so the allowance is not real), pay (the origin answered HTTP 402 Payment Required, meaning it will serve this agent on commercial terms), or unknown. The full definition of each, including what it obliges a crawler to do, is published at https://crawlcensus.com/api/v1/verdicts. Built for crawler operators rather than site owners: it prevents wasted fetches against doors that are shut, and flags content an operator is trying to sell rather than withhold.

输入模式

{'type': 'object', 'required': ['agent', 'domains'], 'properties': {'agent': {'type': 'string', 'description': 'Crawler token, e.g. gptbot, claudebot, perplexitybot, oai-searchbot, ccbot.'}, 'domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Domains to check. Up to 25 per call anonymously; send an Authorization: Bearer key for more. An over-large batch is refused outright rather than partly answered.'}}}

scan_site

Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured data and llms.txt. Returns a score out of 100 with per-check detail.

输入模式

{'type': 'object', 'required': ['domain'], 'properties': {'domain': {'type': 'string', 'description': 'Bare hostname, for example example.com'}}}

site_report

Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.

输入模式

{'type': 'object', 'required': ['domain'], 'properties': {'domain': {'type': 'string', 'description': 'Bare hostname'}}}

submit_domains

Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as unknown with measurable true is worth submitting, and the reply names any that were already fresh or that this census will never measure, so a caller looping over its own unknowns converges instead of resubmitting the same set. Queueing is a database write rather than a fetch, so the allowance is far higher than scan_site and submitted domains are measured ahead of the ranked backlog.

输入模式

{'type': 'object', 'required': ['domains'], 'properties': {'domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Hostnames to queue. Up to 50 per call anonymously; an over-large batch is refused outright rather than partly queued.'}}}

已添加

                            submit_domains
                        

2026年9月17日 12:52

已添加

                            census_facts
                        

2026年9月17日 12:52

已添加

                            agent_profile
                        

2026年9月17日 12:52

已添加

                            crawl_preflight
                        

2026年9月17日 12:52

已添加

                            census_stats
                        

2026年9月17日 12:52

已添加

                            site_report
                        

2026年9月17日 12:52

已添加

                            scan_site
                        

2026年9月17日 12:52

服务器详情

版本

1.0.0

传输方式

Streamable HTTP

工具

延迟

209 ms

端点

https://crawlcensus.com/mcp

近期扫描

成功

2026年10月1日 02:50

成功

2026年9月29日 02:58

成功

2026年9月27日 02:49

成功

2026年9月25日 02:58

成功

2026年9月23日 02:49

成功

2026年9月21日 02:56

成功

2026年9月19日 02:47

成功

2026年9月17日 12:52

Crawl Census

此 MCP 可以做什么

工具

近期工具变更

类似的 MCP 服务器

apis-io

Tools for Agents

canton-ccpedia

webbersites-x402-data-api

api-evangelist

arch-tools-mcp

Agent Margin Router

1cent Web Intelligence for AI Agents