Can AI crawlers read your site?
Enter your website and we read its robots.txt live, then tell you which AI crawlers it lets in and which it turns away. Free, no sign-up.
Why this matters
Blocked means unread
An engine that is told to stay out of your site cannot read your pages, and it is less likely to describe you accurately or cite you as a source.
Often blocked by accident
Many sites block AI crawlers through a template, a plugin or a security default nobody chose on purpose. One line in robots.txt can undo it.
Training and answers are different
Some crawlers collect pages for training, others fetch them to answer a question. You can allow one and block the other; the table shows which is which.
Crawlers are only the first check. The free AI visibility check shows whether engines actually name your business when customers ask.
Run the free visibility checkQuestions people ask
Which crawlers do you check?
GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), PerplexityBot, Google-Extended (Google’s control for Gemini), CCBot (Common Crawl) and Bytespider (ByteDance): the same list AEOHIT’s site audit uses.
Does blocking Google-Extended remove me from AI Overviews?
No. Google-Extended only controls whether Google may use your pages for Gemini. AI Overviews and AI Mode draw on Google Search, which uses Googlebot.
My robots.txt allows everything but AI still ignores me. Why?
robots.txt is not the only gate: Cloudflare’s AI-bot blocking, firewalls and rate limits can stop crawlers too, and this check cannot see those. Being readable is also only the start; engines name businesses that trusted sources mention.
Is this check run live?
Yes. Every check fetches your robots.txt at that moment from our server. We follow one redirect only, from your bare domain to its www version, and we do not store the result.