Skip to content
← Back to Tools

robots.txt AI Crawler Tester

Check which AI crawlers can access your site. Distinguish between AI search crawlers (for answers) and training crawlers (for model training).

§ Methodology

This tool fetches your robots.txt file and tests each known AI crawler user-agent against your rules. It distinguishes between search crawlers (used to retrieve content for answers) and training crawlers (used to collect data for model training).

Version 1.0 · Updated Nov 2026 · Crawlers: Multiple tested

Frequently Asked Questions

Should I block AI search crawlers?

Generally no. Blocking AI search crawlers means your site won't appear in AI-powered answers from ChatGPT, Perplexity, Claude, and similar tools. This is different from training crawlers—blocking those only opts you out of training, not search.

What's the difference between search and training crawlers?

Search crawlers retrieve pages in real-time to answer specific questions. Training crawlers collect large amounts of data to improve the underlying AI models. You can allow search while blocking training.

How do I allow or block specific crawlers?

Edit your robots.txt file. To block a crawler, add User-agent: GPTBot then Disallow: /. To allow one, add User-agent: GPTBot then Allow: /.

Want a complete AI visibility check with automated fixes?

Run Full Scan