Log in
Directory

The AI crawlers visiting your site.

What each crawler does, whether to let it in and how to control it in robots.txt. Search crawlers decide whether your site can appear and be cited in AI answers.

There are three kinds of AI crawlers. Search crawlers read pages to answer and cite sources. User agents open a page when someone asks the assistant to. Training crawlers collect content to train models. To show up in answers, the key is not to block the first two.

Which AI engines rely on these crawlers? AI engines →

Try Neural Ranks

See where your brand stands in AI search.

Enter your site, confirm the suggested topics and see your first real analysis across the 5 AI engines. Free, no card.