Robots.txt Audit Kit — Offline Rules, Warnings, Sitemaps & Can-Fetch Matrix

Sold by TradeSheets with secure USDC checkout.

Description

Offline robots.txt audit for developers and agents: parse local robots.txt files with the MIT robotspy parser (RFC 9309), report every Allow/Disallow rule per crawler group, parser warnings and errors with line numbers, declared sitemaps, and a can-fetch matrix for the user-agents and paths you probe — all without fetching any URL. The batch wrapper audits many files or directories in one run and writes rules.csv, findings.csv, sitemaps.csv, matrix.csv, and summary.json, with inputs hash-verified unchanged. Standard library only, no network, no account. Includes the upstream robotspy tool unchanged with attribution, a batch wrapper, fictional samples, and tests. Crawl-delay and request-rate directives are reported as ignored warnings, not enforced; a clean result describes the file supplied, not what a server serves today.