Blocked by robots.txt: How to Diagnose and Fix the Error
“Blocked by robots.txt” means the crawler was not permitted to fetch that exact URL. Identify the crawler and winning rule before editing anything. If the page belongs in search, narrow the rule, publish the change safely, test the live URL, and allow time for recrawling.
What does “blocked by robots.txt” actually mean?
The status means the named crawler is disallowed from fetching the URL by a rule in robots.txt. It reports a crawl permission, not an indexing directive, server authorization failure, canonical choice, or final search outcome. Start with Google’s robots.txt introduction, not assumptions about the page.
A blocked URL can sometimes appear in Google when external links reveal it, although Google cannot read its content and may lack a useful snippet. Google’s blocked-URL guidance explains this important edge case. Therefore, “blocked” does not guarantee exclusion.
If a public page should be indexed, remove or narrow the applicable disallow rule and verify the result. If it should stay out of search, let crawlers access it and apply noindex as documented in Google’s robots meta guidance. Protect genuinely private content with authentication; robots.txt is public, advisory, and never access control.
How do you confirm the blocked URL and crawler?
Record the exact URL and reported user agent before opening an editor. Include protocol, hostname, subdomain, path, capitalization, query string, and where the warning appeared. A rule affecting https://www.example.com/a does not automatically explain behavior on https://example.com/a or another crawler. Then follow this evidence trail:
- Open the affected origin’s root file, such as
https://example.com/robots.txt; Google specifies the required location in its creation guide. - Use Search Console URL Inspection for the exact URL and review the relevant Page indexing report.
- Check Search Console’s robots.txt report for Google’s fetched version and processing issues.
- Save the visible rule, test result, and timestamp. Do not invent evidence for a property you cannot access.
Which robots.txt rule wins?
Determine the most specific matching user-agent group, then compare every matching Allow and Disallow path. Under RFC 9309, the rule with the longest matching path wins; when equally specific allow and disallow rules conflict, Allow wins. Blank disallows block nothing.
This narrow exception permits one public item beneath a blocked section:
User-agent: Googlebot
Disallow: /collections/private/
Allow: /collections/private/public-itemMatch the URL as the crawler does. Robots rules are scoped to the protocol, host, and port serving the file. Paths are case-sensitive, and percent-encoded characters can change matching. Google documents its supported matching behavior and wildcards in the robots.txt specification.
How do you distinguish robots.txt from other exclusions?
Reproduce both the robots file and target URL because similar symptoms require different fixes. A robots disallow is not a noindex, canonical selection, HTTP 401/403, CDN challenge, or WAF denial. The deciding evidence is the matching directive plus the actual HTTP response observed for the named crawler.
| Evidence | What it indicates | Correct next move |
|---|---|---|
Matching Disallow and reachable target | Robots crawl restriction | Narrow or remove that rule |
noindex in HTML or header | Indexing directive | Keep crawling allowed so it can be read |
| 401/403 or challenge page | Origin, CDN, or WAF access block | Inspect authentication, firewall rules, and logs |
| Alternate canonical | Consolidation signal | Review canonical intent separately |
Test the exact host, subdomain, protocol, case, and encoded path. If robots.txt allows the URL but the server returns 403 to Googlebot, changing robots.txt cannot solve it. Conversely, a 200 response in your browser does not prove Googlebot is allowed.
How should you fix the winning rule safely?
Change only the directive responsible for the unintended block. Broad Allow: / edits can expose crawl traps or sensitive workflows while hiding the original cause. Keep a rollback copy, review the diff, and test the affected URL plus neighboring paths that should remain restricted before deploying.
Use this sequence:
- Replace an overly broad disallow with the narrowest path that expresses the real restriction.
- Preserve unrelated groups and intentional exclusions.
- Deploy through the platform’s supported configuration path.
- Confirm
/robots.txtis public, current, and served successfully as plain text. - If an emergency change is not being picked up, use Search Console’s robots report to request a recrawl of the file; this is a request, not an instant refresh promise.
- Run a live URL test, then wait and monitor crawl and indexing reports.
Do not promise immediate crawling or indexing. Google schedules retrieval, processing, and indexing independently, and a technically crawlable page can still remain unindexed for quality, duplication, canonicalization, or other reasons.
How do you customize robots.txt safely on Shopify?
Shopify generates a default robots.txt that suits most stores. When customization is necessary, use templates/robots.txt.liquid, retain robots.default_groups, and append only targeted rules. Shopify can update its defaults, so replacing the generated groups creates maintenance debt and can silently discard future platform improvements.
Follow Shopify’s official robots.txt.liquid template guidance and robots Liquid object reference. Never try to edit a static file at the site root. Do not broadly allow cart, checkout, customer-account, or internal-search paths merely to clear one warning.
Make theme or app changes reversible, preview the rendered output, and production-test both desktop and mobile storefront paths. A Shopify collection or product blocked accidentally deserves a precise exception; system routes should not be reopened as collateral damage.
What should you verify after deployment?
Verification must prove the public rule changed and the target became fetchable for the intended crawler; a successful theme save or HTTP 200 alone is insufficient. Capture the rendered robots file, response headers, live test result, affected URL, deployment identifier, and timestamp so another person can reproduce the conclusion.
Monitor Search Console over time rather than repeatedly changing the file. A live test can confirm current access, while reporting may reflect an earlier crawl. StoreCited offers a public, point-in-time crawler readiness check and an AI crawler checker, but it has no Search Console access by default, cannot see private server logs, and cannot automatically fix or guarantee indexing.
Does a Googlebot block also block AI crawlers?
No. A rule for Googlebot is not proof that OAI-SearchBot, GPTBot, ClaudeBot, or PerplexityBot is allowed or blocked. Each product uses its own documented user-agent behavior, and CDN or WAF policy can differ from the public robots file. Evaluate each crawler independently without losing focus on the Google diagnosis.
For OpenAI crawlers, consult the official OpenAI bots documentation. For every provider, inspect its current official user-agent policy, the matching group in the live robots file, the target response, and actual firewall or server logs when available. Absence from logs is not proof of permission or attempted crawling.
Get the answer for your specific store