A LITTLE HELP FOR YOUR NEXT TASK
Robots.txt Rule Tester
Test crawl permissions with matching rules and user-agent precedence.
Private by design
Your inputs stay on your device and are not sent to our servers. Optional usage analytics does not receive your files or tool inputs.
Preparing web and network workspace…
How to use Robots.txt Rule Tester
- Add your source or choose Try sample; Undo sample restores previous inputs.
- Choose the workflow and review results, diagnostics and method notes.
- Download the useful artifact, complete records or a summary report.
What are the limits of Robots.txt Rule Tester?
Local Google-style robots matching: most-specific matching user-agent groups merge, longest matching path wins, Allow wins ties, * and trailing $ are supported. UTF-8 percent-encoding is normalized with reserved octets preserved. No live crawl, robots download, indexing guarantee or support for crawl-delay. Robots exclusions are not authorization controls. Text input: 5 MB total. XML external entities and includes are disabled. No automatic local storage. Reports summarize the selected result; CSV/JSON retain complete records.