OAI-SearchBot
OAI-SearchBot is OpenAI’s crawler used for ChatGPT search: the pages ChatGPT can show and cite in its answers.
What it means for your site
It reads pages to answer and cite sources. If you block it, your site can no longer appear or be cited in those answers.
Should you allow or block it?
Allow it if you want your brand to appear and be cited in AI answers. It’s the most important crawler for AI visibility.
How to allow or block it
To allow it (in the robots.txt file at the root of your site):
User-agent: OAI-SearchBot Allow: /
To block it:
User-agent: OAI-SearchBot Disallow: /
Anyone can put a crawler’s name in their user agent. To confirm a visit is genuine, check the IP address against the list the company publishes. Blocks at the CDN or firewall don’t show up in robots.txt.
How to know if it visits your site
The robots.txt checker shows whether this crawler is allowed. With a Neural Ranks account, the AI crawlers page in your dashboard shows how often each crawler visits your site and which pages it reads.
Other AI crawlers
AI engines that rely on this crawler: ChatGPT
See where your brand stands in AI search.
Enter your site, confirm the suggested topics and see your first real analysis across the 5 AI engines. Free, no card.