Home / Servers / Developer Toolbox / generate_robots_txt
Generate robots txt
An MCP tool from the Developer Toolbox server, hosted on mcp.hjlabs.in.
Compose a robots.txt with allow/disallow rules, an optional crawl delay, a sitemap reference, and an explicit choice about AI crawlers (allow, block, or leave unmentioned).
Name on the gateway: devtools__generate_robots_txt · read-only.
Parameters
| Name | Type | Description |
|---|---|---|
sitemapUrl | string | Absolute URL of the sitemap. |
allow | array | Paths to allow. |
disallow | array | Paths to disallow. |
crawlDelay | number | Seconds between requests. |
aiCrawlers | allow | block | omit | Default "allow". |
How to call it
Through the Smart Gateway, any client can run it by name with call_tool, or let the gateway find it with search_tools:
{
"name": "devtools__generate_robots_txt",
"arguments": {}
}Or connect the Developer Toolbox server directly, which lists 18 tools including this one:
claude mcp add --transport http hj-devtools https://mcp.hjlabs.in/mcp/devtools \
--header "Authorization: Bearer YOUR_API_KEY"Try asking
Use the generate robots txt tool with sitemapUrl set to ….