What this MCP does
Checks AI crawler permissions and visibility files, explains crawler behavior, and shows how pages appear when fetched without JavaScript.
Tools
Input schema
{'type': 'object', 'required': ['url'], 'properties': {'url': {'type': 'string', 'description': 'Website or page address, e.g. example.com or https://example.com/pricing'}}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['url', 'robotsTxt', 'agents'], 'properties': {'url': {'type': ['string', 'null']}, 'agents': {'type': 'array', 'items': {'type': 'object', 'required': ['token', 'verdict', 'operator', 'role'], 'properties': {'role': {'enum': ['search', 'user', 'training', 'control'], 'type': 'string'}, 'token': {'type': 'string'}, 'verdict': {'enum': ['allowed', 'default', 'blocked'], 'type': 'string'}, 'operator': {'type': 'string'}}}}, 'llmsTxt': {'type': ['boolean', 'null']}, 'sitemaps': {'type': 'array', 'items': {'type': 'string'}}, 'robotsTxt': {'enum': ['ok', 'missing', 'unreachable', 'invalid', 'pasted'], 'type': 'string'}, 'httpStatus': {'type': ['integer', 'null']}, 'contentSignals': {'type': 'array', 'items': {'type': 'string'}}}}
Input schema
{'type': 'object', 'properties': {'name': {'type': 'string', 'description': 'Crawler user agent, e.g. GPTBot. Omit to list all.'}}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['crawlers'], 'properties': {'crawlers': {'type': 'array', 'items': {'type': 'object', 'required': ['token', 'operator', 'role', 'does'], 'properties': {'does': {'type': 'string'}, 'role': {'type': 'string'}, 'token': {'type': 'string'}, 'product': {'type': 'string'}, 'operator': {'type': 'string'}, 'roleLabel': {'type': 'string'}}}}}}
Input schema
{'type': 'object', 'properties': {'company': {'type': 'string', 'description': 'Company name, e.g. Notion. Omit to list all reports.'}}, 'additionalProperties': False}
Output schema
{'type': 'object', 'properties': {'url': {'type': 'string'}, 'grade': {'type': 'string'}, 'score': {'type': 'integer'}, 'report': {'type': 'string'}, 'company': {'type': 'string'}, 'pillars': {'type': 'array', 'items': {'type': 'object', 'properties': {'id': {'type': 'string'}, 'label': {'type': 'string'}, 'score': {'type': 'integer'}, 'notAssessed': {'type': 'boolean'}}}}, 'reports': {'type': 'array', 'items': {'type': 'object', 'properties': {'url': {'type': 'string'}, 'grade': {'type': 'string'}, 'score': {'type': 'integer'}, 'company': {'type': 'string'}, 'scannedAt': {'type': 'string'}}}}, 'scannedAt': {'type': 'string'}}, 'description': 'Either `reports` (a list, when no company matched) or the single report fields.'}
Input schema
{'type': 'object', 'required': ['url'], 'properties': {'url': {'type': 'string', 'description': 'Website or page address, e.g. example.com or https://example.com/pricing'}}, 'additionalProperties': False}
Output schema
{'type': 'object', 'required': ['finalUrl', 'status', 'words', 'fullScan'], 'properties': {'h1': {'type': 'array', 'items': {'type': 'string'}}, 'bytes': {'type': 'integer'}, 'title': {'type': ['string', 'null']}, 'words': {'type': 'integer'}, 'status': {'type': 'integer'}, 'finalUrl': {'type': 'string'}, 'fullScan': {'type': 'string'}, 'canonical': {'type': ['string', 'null']}, 'htmlTokens': {'type': 'integer'}, 'textTokens': {'type': 'integer'}, 'headingCount': {'type': 'integer'}, 'structuredData': {'type': 'array', 'items': {'type': 'string'}}, 'metaDescription': {'type': ['string', 'null']}}}
Recent tool changes
Similar MCP servers
TinyFn
Offers deterministic utility functions for mathematics, conversions, validation, hashing, encoding, arrays, dates, colors, and pa…
hyperion
Acts as a paid MCP tool marketplace and utility gateway with server discovery, HTTP and JavaScript tools, research, data conversi…
Andreax
Offers pay-per-call AI services for inference, agent and workflow design, OCR and transcription, code generation and review, clas…
Vee3
Manages Clerk authentication infrastructure, including users, organizations, domains, sessions, tokens, OAuth, SSO, machines, per…
GripForge
Generates, rigs, animates, analyzes, and packages game characters, weapons, armor, effects, environments, and engine-ready assets.
Courier
Provides notification delivery infrastructure for users, tenants, lists, templates, preferences, journeys, automations, brands, a…
IA-QA — 130+ QA & Dev Tools for AI Agents
Provides deterministic QA, evaluation, testing, code analysis, prompt and RAG checks, model comparison, and web security diagnost…
apis-io
Provides catalog search, comparison, scoring, enrichment, lists, and dataset exports for APIs, providers, specifications, workflo…