Products → Robots, Sitemap & Crawl-Directives Skill Pack for Claude Code
Robots, Sitemap & Crawl-Directives Skill Pack for Claude Code
A Claude Code skill pack that parses robots.txt the way real crawlers do — precedence, wildcards, multiple user-agent groups — validates your sitemap's structure, and cross-references the two: pages disallowed but still sitemapped, noindexed or 404'd sitemap entries, crawl-budget waste, orphaned pages. Plus a pass/fail guard for CI. Runs in your editor, no dependencies, no SaaS, €49 once.
v1.0.0 — live. The checker was run against real live sites at real scale: github.com's actual multi-agent wildcard robots.txt and wordpress.org's real sitemap index (27 files, 2,499 URLs, one genuine conflict found). The worked outputs ship in the download (esempio/).
The problem this solves
robots.txt and sitemap.xml are usually written by different people at different
times, and nothing checks that they still agree. A Disallow rule
meant for a staging path also matches a production URL that's proudly listed in
your sitemap — search engines get told "crawl this" and "don't crawl this" in the
same breath. Sitemaps quietly list pages that 404, redirect, or carry a
noindex. Multiple User-agent groups each have their own
wildcard rules, and nobody's sure which one actually applies. A sitemap index with
20+ child files can duplicate the same URLs across shards, or quietly exceed the
50,000-URL limit on one of them. You could open both files side by side and check
by hand. Nobody does it after the first launch.
What's inside
| Skill | What it does |
|---|---|
crawl-directives-audit | Parses robots.txt with the precedence rules real crawlers use (most-specific-path wins, Allow beats Disallow on a tie, */$ wildcards, multiple user-agent groups), validates sitemap.xml/sitemap-index structure and limits, and cross-references every sitemap URL against robots.txt and — with a crawl.json — actual page status, noindex, and canonical. Flags crawl-budget waste and orphaned pages too. |
robots-sitemap-guard | The same checks as a binary pass/fail guard: --assert exits non-zero on a regression. Validates a draft robots.txt/sitemap.xml before it ships, or the live site after, so a bad Disallow rule can't reach production silently. |
Plus two dependency-free helper scripts: robots-sitemap-check.mjs (the parser + validator + cross-reference engine) and crawl.mjs (the raw-HTML crawler shared with the other mulino.tools packs), an install guide, and worked examples against real live sites.
What it is not
- It does not rewrite robots.txt for you. It finds the conflict and names the fix; editing the file is yours.
- It is not a guess machine. Every finding traces to a value it actually parsed or fetched — no invented crawl-budget numbers.
- It is not a SaaS. No account, no API key, no monthly fee, no npm dependencies. Files in a folder.
Why €49
If your time is worth €50 an hour, the pack pays for itself the first time it catches one robots.txt rule that's silently blocking sitemapped pages — the kind of bug that costs weeks of "why isn't this indexed" before anyone thinks to diff the two files by hand. Then it keeps checking on every deploy, which nobody was going to do manually.
What you get
- 2 skills and 2 dependency-free helper scripts
- Install guide (~2 minutes) and a QA checklist showing how each skill was tested
- Worked examples: github.com's real multi-agent wildcard robots.txt, wordpress.org's real sitemap index at 2,499-URL scale, a broken/fixed draft pair, and a labelled synthetic fixture exercising every cross-reference check
- 12 months of updates — a bounded, honest commitment, not "lifetime"
- 30-day refund, no questions asked, through Gumroad
Requirements
- Claude Code (free)
- Node.js 18+. No npm dependencies, no API keys, no account.
- robots.txt/sitemap.xml reachable over HTTP(S), or local draft files for pre-deploy validation.
FAQ
- Isn't this what Google Search Console's Sitemaps report already shows?
- Search Console tells you a sitemap was processed and how many URLs got indexed — it doesn't tell you why a specific URL didn't, or that robots.txt is the reason. This finds the exact rule and the exact URL, before or after you deploy.
- Does it handle a sitemap index with multiple child sitemaps?
- Yes — it recurses into a
sitemapindexautomatically. Tested at real scale (27 child sitemaps, 2,499 URLs, one run) against wordpress.org. - Which robots.txt rule wins when several match a URL?
- The two rules that decide real-world outcomes: the most specific (longest)
matching path wins, and Allow beats Disallow on an exact tie.
*and$are supported — not a claim to implement every crawler's undocumented edge case, the part that actually decides outcomes in practice. - Can I check a robots.txt before it's even deployed?
- Yes — local draft files can be checked directly, so the guard can run against your build output in CI before anything ships.
- Do I need to know how to code?
- No. You need Claude Code and Node.js. You describe the site in plain English and the skills run the script and read the output.
- Will you keep it updated?
- Updates for 12 months from purchase, through your Gumroad library and logged in a changelog. After that the version you have keeps working; further updates are not included. Deliberately not a "lifetime" promise.
AI disclosure
The skills are written instructions for an AI coding assistant and were authored with AI assistance. The helper scripts are conventional code and report only measured values — no invented metrics, ratings, or stats anywhere in the product or on this page.
Fourth in the mulino.tools technical-SEO set for Claude Code: the Technical SEO Skill Pack audits and fixes an existing site, the Site Migration & Redirects Skill Pack covers the replatform/relaunch moment, the Rendering & JavaScript SEO Skill Pack covers the raw-vs-rendered gap JS frameworks introduce, and this one covers robots.txt vs sitemap.xml, plus a CI guard for both.