Home / Servers / Developer Toolbox / generate_ai_robots_txt
Generate ai robots txt
An MCP tool from the Developer Toolbox server, hosted on mcp.hjlabs.in.
Compose robots.txt rules for the known AI crawlers with a single allow-or-block policy plus per-agent exceptions, and optional Content-Signal directives. Annotates what each agent is for.
Name on the gateway: devtools__generate_ai_robots_txt · read-only.
Parameters
| Name | Type | Description |
|---|---|---|
policy | allow | block | The default for AI crawlers. |
exceptions | array | Agent names to flip against the default. |
contentSignal | boolean | Include Content-Signal directives. Default true. |
sitemapUrl | string |
How to call it
Through the Smart Gateway, any client can run it by name with call_tool, or let the gateway find it with search_tools:
{
"name": "devtools__generate_ai_robots_txt",
"arguments": {}
}Or connect the Developer Toolbox server directly, which lists 18 tools including this one:
claude mcp add --transport http hj-devtools https://mcp.hjlabs.in/mcp/devtools \
--header "Authorization: Bearer YOUR_API_KEY"Try asking
Use the generate ai robots txt tool with policy set to ….