Does your site let AI in?
Enter your domain. We read the robots.txt and show which AI crawlers are allowed or blocked, and whether you have a sitemap and an llms.txt.
The AI crawlers we check.
Blocking training crawlers doesn’t remove your site from answers. Blocking search crawlers does.
| Crawler | Company | What it does |
|---|---|---|
GPTBot | OpenAI | Model training |
OAI-SearchBot | OpenAI | AI search and answers |
ChatGPT-User | OpenAI | User-requested visits |
ClaudeBot | Anthropic | Model training |
Claude-SearchBot | Anthropic | AI search and answers |
Claude-User | Anthropic | User-requested visits |
PerplexityBot | Perplexity | AI search and answers |
Perplexity-User | Perplexity | User-requested visits |
Googlebot | AI search and answers | |
Google-Extended | Model training | |
Bingbot | Microsoft | AI search and answers |
Applebot-Extended | Apple | Model training |
Amazonbot | Amazon | AI search and answers |
meta-externalagent | Meta | Model training |
CCBot | Common Crawl | Model training |
Bytespider | ByteDance | Model training |
Training, search and visits are not the same
Each AI company uses several crawlers. Training crawlers, such as GPTBot, ClaudeBot and Google-Extended, collect content to train models. Search crawlers, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, read pages to answer questions and cite sources. User agents, such as ChatGPT-User, open a page when someone asks the assistant to.
To appear in answers, the key is not to block search and user crawlers, nor Googlebot and Bingbot, which feed AI Overviews, Copilot and ChatGPT search. Blocking only training crawlers is a legitimate choice and usually doesn’t reduce your presence in answers.
How to fix a block
In the robots.txt file at your site root, add a group for the crawler and an Allow: / line. For example: User-agent: OAI-SearchBot followed by Allow: /. Also check that your CDN or firewall isn’t blocking these crawlers, because that kind of block doesn’t show up in robots.txt.
This tool only reads the public robots.txt and uses the home page as the test path. It doesn’t detect blocks made by a firewall, CDN or the server.
See where your brand stands in AI search.
7 days free, no card. Your analysis, your competitors and a sample action plan.