Free tool
AI Crawler Access Checker (robots.txt)
Inspect what robots.txt declares for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended.
Robots rules can block a named crawler from fetching public pages, but crawler purposes differ and access never guarantees use or citation. Check retrieval and training/data-use user agents separately, plus whether the site publishes an optional llms.txt.
Frequently asked questions
Should I allow AI crawlers like GPTBot?
Decide by user-agent purpose. OAI-SearchBot and PerplexityBot support retrieval, while GPTBot and CCBot are associated with training or data collection. Google-Extended controls certain Google AI training and grounding uses, not Google Search indexing. Access choices are separate from citation guarantees.
What is llms.txt?
llms.txt is an emerging, optional plain-text convention for summarizing important pages and facts. Support varies. It does not replace robots.txt, a sitemap, crawlable HTML, Shopify Catalog, or any platform's documented product-feed mechanism.
How accurate is this check?
It reads the fetched robots.txt and applies standard matching rules to the listed user agents. That verifies declared access at one point in time; it cannot prove identity, successful rendering, indexing, retrieval, recommendation, or citation.
More free tools
Want the full picture, not just one signal?
Run a free scan for a point-in-time public-storefront readiness score, peer hypotheses to verify, and prioritized audit checks.
Run free scan