By Team Edtoolkits · Web & network
Allow or block GPTBot, ClaudeBot, Google-Extended and other AI crawlers in robots.txt, with a plain explanation and official source for each.
Use the guide alongside the working tool.
Open AI Crawler Rules Builder →Get started
Pick a quick stance or set each crawler to Allow, Block or Leave alone.
Adjust and explore
Add your sitemap URLs, if you have them.
Save and share
Copy the robots.txt or download it and upload it to your site root.
Privacy and limits
Robots.txt is a request, not access control, and some fetchers triggered by a person may ignore it. Tokens were checked against each operator's documentation in November 2026; operators change them, so recheck before relying on it. Your data stays on your device.
Ready to apply this workflow?
Try AI Crawler Rules Builder →Explore a related workflow
- DNS Record Inspector: practical guide
Look up public DNS records and inspect resolver answers.
- DNS Zone File Validator: practical guide
Check common BIND zone records and configuration conflicts.
- SPF Record Analyzer: practical guide
Review SPF mechanisms, IP ranges and DNS lookup limits.
- DMARC Record Studio: practical guide
Build and check a DMARC DNS policy with clear tag explanations.
Keep your original files and review the result. Found unclear instructions? Contact Team Edtoolkits.