Home / Servers / Web and Network / robots_check
Robots check
An MCP tool from the Web and Network server, hosted on mcp.hjlabs.in.
Fetch a site's robots.txt and say whether a given path may be crawled by a given user agent (longest-match rules, RFC 9309), plus the sitemaps it declares and the user-agent groups it defines. Use it before scraping, or to debug why a page is not indexed.
Name on the gateway: web__robots_check · read-only.
Parameters
| Name | Type | Description |
|---|---|---|
url required | string | Any URL on the site; the path part is what gets checked (default "/"). |
user_agent | string | Crawler name to evaluate, e.g. "Googlebot" or "*". Defaults to "*". |
How to call it
Through the Smart Gateway, any client can run it by name with call_tool, or let the gateway find it with search_tools:
{
"name": "web__robots_check",
"arguments": {
"url": "…"
}
}Or connect the Web and Network server directly, which lists 22 tools including this one:
claude mcp add --transport http hj-web https://mcp.hjlabs.in/mcp/web \
--header "Authorization: Bearer YOUR_API_KEY"Try asking
Use the robots check tool with url set to ….