anthropic-ai
anthropic-ai is Anthropic’s crawler used for an old name used by Anthropic, still found in many robots.txt files.
What it means for your site
It’s an old name, replaced by other crawlers from the same company. It still appears in many robots.txt files.
Should you allow or block it?
You can keep your existing rule. Make sure the current crawlers from the same company have consistent rules.
How to allow or block it
To allow it (in the robots.txt file at the root of your site):
User-agent: anthropic-ai Allow: /
To block it:
User-agent: anthropic-ai Disallow: /
Anyone can put a crawler’s name in their user agent. To confirm a visit is genuine, check the IP address against the list the company publishes. Blocks at the CDN or firewall don’t show up in robots.txt.
How to know if it visits your site
The robots.txt checker shows whether this crawler is allowed. With a Neural Ranks account, the AI crawlers page in your dashboard shows how often each crawler visits your site and which pages it reads.
See where your brand stands in AI search.
Enter your site, confirm the suggested topics and see your first real analysis across the 5 AI engines. Free, no card.