Skip to content
Check my business

AI crawlers

AI crawlers are the bots AI companies use to fetch web pages, such as OpenAI’s GPTBot, Anthropic’s ClaudeBot and PerplexityBot; Google-Extended is a robots.txt token controlling use of content for Google’s AI models.

How they are controlled

Each crawler announces a user-agent name, and your robots.txt can allow or block it by that name. Google-Extended is not a separate crawler: it is a token Google reads to decide whether your content may be used for its AI models.

Some companies run more than one bot with different jobs, for example OpenAI documents GPTBot for training and OAI-SearchBot for search, so blocking one does not necessarily block the other.

Why it matters for a business

A robots.txt rule written years ago, or added by a plugin, can quietly block engines that would otherwise read and recommend you.

How AEOHIT helps

The free AI crawler check reads your robots.txt and shows which AI crawlers it allows or blocks. AEOHIT plans write the exact lines to change.

AI crawlers: common questions

Should I block AI crawlers?

It is your choice. Blocking them can keep your content out of training or answers, but engines that cannot read your site are less likely to recommend you.

Does blocking GPTBot remove me from ChatGPT search?

Not necessarily. OpenAI documents separate bots for training (GPTBot) and search (OAI-SearchBot), each controlled by its own robots.txt rule.

See what AI engines say about your business.

Free report in about a minute. No card, no sign-up.

Check my business