Should You Block AI Crawlers Like GPTBot?

Should You Block AI Crawlers Like GPTBot?

No. If you're a Shopify or DTC store that wants to be found and cited by AI, blocking crawlers like GPTBot, PerplexityBot, or ClaudeBot is close to the worst move you can make. These bots are how AI systems learn your store exists, verify your prices, and decide whether to recommend you over a competitor. Block them, and you've opted out of a channel you can't opt back into instantly — the next crawl might be weeks away.

This isn't a neutral "it depends" answer. It's a stance: for a store, the downside of blocking is real and immediate (invisibility), while the downside of allowing is mostly theoretical (a vague worry about "AI stealing my content"). Your product descriptions, prices, and return policy aren't proprietary IP in a way that matters commercially — being found is what matters.

Why Some Store Owners Consider Blocking AI Crawlers

The instinct to block usually comes from one of three worries: content scraping for AI training, server load, or a general "I don't understand what this bot is doing, so I'll say no" caution. All three are worth addressing directly rather than dismissing.

• "They're training AI on my content for free." True for training-only crawlers like GPTBot and CCBot. But your product pages already exist to be read, indexed, and referenced — that's the entire point of publishing them on the open web.
• "It's extra server load." Legitimate AI crawlers are far lighter than a single Google Ads landing-page spike. This is rarely a real capacity issue for a Shopify store, which runs on Shopify's own infrastructure, not yours.
• "I don't know what half these bots do." Fair — the AI crawler landscape moved fast in 2024-2025 and most store owners have never had to think about user-agents before. That's exactly why the list below exists.

None of these concerns outweigh the cost of being unreachable when someone asks ChatGPT or Perplexity "what's a good [your category]?" and your store never surfaces because it was never crawled. If you're weighing "should I block AI crawlers" against a specific worry, run it through the free AI search visibility checker first — most stores that think they have a content-protection reason discover they're already fully indexed by regular search anyway.

The Two Types of AI Bots — And Why the Distinction Matters

AI crawlers fall into two functional categories, and conflating them is the most common mistake in this decision. Retrieval bots fetch your page live to answer a specific question right now; training bots scrape broadly to build a future model. Your allow/block decision should differ by category, not be all-or-nothing.

If you're going to make any distinction at all, retrieval bots are non-negotiable — blocking OAI-SearchBot or PerplexityBot directly removes you from citation eligibility for that query, today. Training bots are a softer, longer-horizon call. Even so, Google's own AI features documentation treats broad crawlability as foundational to appearing in AI Overviews — there's no separate "AI Overviews crawler" you can selectively allow while blocking everything else.

The Full List of AI Crawlers You Should Know

Here's the practical list of major AI user-agents active as of early 2026, what they're for, and StoreCited's recommendation for each. Check any of these against your live robots.txt with the free AI Crawler Checker rather than guessing.

• GPTBot — OpenAI's general training crawler. Allow, unless you have a specific content-licensing objection.
• OAI-SearchBot — Powers ChatGPT's live web search and citations. Allow — this is retrieval, not training.
• ChatGPT-User — Fires when a ChatGPT user asks it to browse a specific page live. Allow.
• ClaudeBot / anthropic-ai — Anthropic's crawlers, feeding Claude's knowledge and citations. Allow.
• PerplexityBot — Perplexity's answer-engine crawler; directly responsible for citations shown in Perplexity results. Allow.
• Google-Extended — Controls whether Google's Gemini models and AI Overviews can use your content beyond standard Search indexing. Allow if you want AI Overviews visibility.
• CCBot — Common Crawl, an open dataset that feeds many different LLMs (not just one company's). Allow for broad AI reach.
• Applebot-Extended — Governs Apple Intelligence's use of your content. Allow.
• Amazonbot — Amazon's crawler, tied to Alexa and shopping features. Allow, especially relevant for DTC/ecommerce.

This isn't an exhaustive registry — new bots appear as AI companies ship new products — which is exactly why a blanket "allow AI crawlers, disallow only private routes" policy is easier to maintain than a hand-picked list you'll forget to update.

How to Set This Up: The robots.txt Approach That Actually Works

The simplest, most durable robots.txt policy is to allow everything by default and explicitly disallow only the private, non-content paths every store already has — admin, cart, checkout, account, and search-results pages. You do not need a hand-curated allow-list per bot; that approach breaks the moment a new AI crawler launches and you haven't added it yet.

A working pattern looks like this:

User-agent: *
Allow: /
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search

Sitemap: https://yourstore.com/sitemap.xml

Because the wildcard User-agent: * group already allows everything except those private paths, every AI crawler — current and future — inherits that permission automatically. You only need bot-specific blocks if you actively want to exclude one specific crawler (say, for a genuine content-licensing dispute), which is the exception, not the rule.

A few practical notes:

• Check your theme and apps, not just robots.txt. Some Shopify SEO apps or third-party page builders inject their own meta robots tags, and some CDN/firewall dashboards now ship a one-click "block AI bots" toggle that store owners enable without realizing the visibility cost. Your structured data and robots.txt settings both need checking — one controls whether a bot can arrive, the other whether it understands what it finds.
• robots.txt is a request, not a lock. Well-behaved crawlers (OpenAI, Anthropic, Google, Perplexity all publish their user-agents publicly) respect it. It won't stop bad-faith scrapers — that's a separate problem, usually solved with rate limiting or a WAF, not with blanket AI-bot bans that also hurt legitimate ones.
• Pair it with an llms.txt. Once crawlers can reach you, llms.txt gives them a concise map of what matters most on your site — your key pages, policies, and facts — in plain text they can parse without guessing. Robots.txt controls access; llms.txt improves comprehension once they're in.

What Actually Happens If You Block Them

Blocking AI crawlers doesn't make your store safer, faster, or more private — it makes you absent from a conversation that's already happening about your category. The realistic outcomes:

1. You lose citation eligibility immediately for any live-retrieval query (ChatGPT search, Perplexity, AI Overviews) the moment the block goes live — no grace period.
2. Competitors who allow crawlers fill the gap. AI answer engines still need to recommend something for "best [your category]" — if you're unreachable, a competitor becomes the default citation.
3. You don't get the content-protection benefit you were hoping for. Your prices, descriptions, and policies are already public on the open web; blocking one class of bot doesn't meaningfully change your IP exposure.
4. Un-blocking doesn't undo the gap instantly. Re-crawling and re-indexing takes time, so a "block now, allow later" strategy costs you real visibility in the interim.

Bottom Line

StoreCited's position is direct: allow the major AI crawlers, disallow only your genuinely private paths, and pair it with an llms.txt so bots that can reach you also understand you quickly. This is a five-minute robots.txt edit with a real ongoing payoff, versus a block that trades a hypothetical concern for a concrete loss of visibility.

Crawlability is step one, not the whole job — being allowed in doesn't guarantee you're structured well enough to be cited once a bot arrives. That's the harder, more valuable half of answer engine optimization: clean structured data, real buyer-question content, and comparison pages an AI can quote with confidence.

Want to see exactly where your store stands on both halves — crawler access and content readiness? Run a free StoreCited scan and get your AI Visibility Score, the competitors AI is citing instead of you, and the specific gaps to fix first. Curious what the fixed version looks like? See how it works.