Directory
The AI crawlers visiting your site.
What each crawler does, whether to let it in and how to control it in robots.txt. Search crawlers decide whether your site can appear and be cited in AI answers.
There are three kinds of AI crawlers. Search crawlers read pages to answer and cite sources. User agents open a page when someone asks the assistant to. Training crawlers collect content to train models. To show up in answers, the key is not to block the first two.
Search and answers
OAI-SearchBotOpenAI · ChatGPT search: the pages ChatGPT can show and cite in its answersClaude-SearchBotAnthropic · Claude search: the pages Claude can use and cite in its answersPerplexityBotPerplexity · Perplexity’s search index: the pages Perplexity can show and citeGooglebotGoogle · Google Search, including AI Overviews and AI ModeBingbotMicrosoft · Bing search, which also powers Microsoft Copilot and other AI toolsApplebotApple · Apple search: Siri, Spotlight and Safari suggestionsAmazonbotAmazon · Amazon services, including Alexa answers
User requests
ChatGPT-UserOpenAI · the visits ChatGPT makes to a page when a user asks it toClaude-UserAnthropic · the visits Claude makes to a page when a user asks it toPerplexity-UserPerplexity · the visits Perplexity makes to a page when a user asks it somethingMistralAI-UserMistral AI · the visits Mistral AI’s Le Chat makes to a page when a user asks it to
Model training
GPTBotOpenAI · training OpenAI’s AI models (GPT)ClaudeBotAnthropic · training Anthropic’s AI models (Claude)Google-ExtendedGoogle · permission for Google to use content to train and ground Gemini (it is not a separate crawler and does not affect Google Search)Applebot-ExtendedApple · permission for Apple to use content to train Apple Intelligence models (it is not a separate crawler)meta-externalagentMeta · training Meta’s AI models and other uses of web contentCCBotCommon Crawl · Common Crawl, a public web archive used to train many AI modelscohere-aiCohere · Cohere’s AI productsBytespiderByteDance · training ByteDance’s AI models (TikTok’s owner)
Which AI engines rely on these crawlers? AI engines →
Try Neural Ranks
See where your brand stands in AI search.
Enter your site, confirm the suggested topics and see your first real analysis across the 5 AI engines. Free, no card.