AI Crawler Access Checker (robots.txt)
See whether GPTBot, ClaudeBot, PerplexityBot, and Google-Extended can crawl your store.
If your robots.txt blocks AI crawlers, you can't be cited no matter how good your content is. Enter your domain to see which AI bots are allowed or blocked — and whether you publish an llms.txt.
Frequently asked questions
Should I allow AI crawlers like GPTBot?
If you want to appear in AI answers and AI search, yes — retrieval-focused bots (OAI-SearchBot, PerplexityBot, Google-Extended) need access to cite you. Some stores block training-only bots (GPTBot, CCBot) as a content-rights choice; that's a strategic decision, not a technical requirement.
What is llms.txt?
llms.txt is an emerging plain-text file at your site root that tells AI crawlers what's most important on your site — a concise map of your key pages and facts. It's optional but increasingly recommended for AI visibility.
How accurate is this check?
It reads your live robots.txt and applies the standard rules to each AI user-agent. It's a fast, accurate read of access — but being crawlable is only step one; your content also has to be structured so AI can understand and cite it.
More free tools
Want the full picture, not just one signal?
Run a free scan for your AI Visibility Score, the competitors AI cites instead of you, and every gap to fix.
Run free scan