How to Use the AI Crawler Checker
- Enter your domain or a specific page URL. For a page URL, rules are evaluated for that exact path.
- Check the summary, which shows how many AI search and assistant crawlers can reach the page, and whether you have robots.txt, llms.txt and an XML sitemap.
- Review the crawler table. Each bot shows its company, what it’s used for, its type, whether it’s allowed, and the exact rule that decided it.
- Copy the suggested rules if any AI search crawlers are blocked and you want them to have access.
The Crawlers We Check
| Company | Search / user crawlers | Training crawlers |
|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | |
| Googlebot (Search, AI Overviews, AI Mode) | Google-Extended | |
| Microsoft | Bingbot (Bing, Copilot) | |
| Apple | Applebot | Applebot-Extended |
| Others | Amazonbot, DuckAssistBot | meta-externalagent, CCBot, Bytespider |
Search vs. Training: Why the Difference Matters
Blocking a training crawler stops your content being collected to train future models, but it doesn’t remove you from AI search results. Blocking a search or user crawler does, because the assistant can no longer read your pages when answering questions.
For most businesses that want customers to find them through AI assistants, a sensible default is:
- Allow search and user crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Bingbot).
- Decide on training crawlers based on how you feel about your content being used for model training.
Common Mistakes
- Blocking everything with one rule. A blanket “Disallow: /” under “User-agent: *” blocks every compliant crawler, including Google.
- Copying a robots.txt from another site that blocks AI crawlers you actually want.
- Forgetting CDN settings. Some hosting and CDN providers offer one-click AI bot blocking that overrides an otherwise open robots.txt.
- Blocking CSS and JavaScript folders, which can stop crawlers rendering your pages correctly.
Next Steps
Once crawlers can reach your site, give them a clear summary of your business with an llms.txt file, and check the rest of your AI readiness with our AI Visibility Checker. For a complete strategy, see our AI Search Optimization service.