SemrushBot Explained: User Agents, Crawl Control, and Blocking

How can you identify Semrush traffic with evidence?

Identify a Semrush bot from the exact user-agent string, then correlate timestamps, requested paths, response codes, request frequency, and any known audit schedule. A name alone is a claim supplied by the requester, not proof of identity. Never report that a crawl occurred unless your own CDN, load-balancer, or origin logs contain matching requests.

For a 10–15 minute window, group requests by exact agent; record host, timestamp, path, status, bytes, latency, and cache result; then compare timing with configured Semrush projects. Repeat the check for each subdomain.

Semrush advises using robots controls instead of IP blocking because it does not publish crawler addresses as stable consecutive ranges. The official crawler page is therefore more reliable for control syntax than an invented allowlist. Reverse DNS can contribute evidence, but it is not guaranteed Semrush authentication.

When should you allow, throttle, or block it?

Allow a named crawler when its product output is useful and resource impact is low. Throttle it when the purpose is legitimate but request rate strains infrastructure. Block only the exact unnecessary agent or sensitive path. This least-disruptive policy preserves audits and research while avoiding accidental damage to unrelated search and AI discovery systems.

• Allow: expected projects and low-cost public crawling.
• Throttle: a useful agent producing measured origin strain.
• Block: an exact unneeded agent or irrelevant public route.

Robots rules affect crawl access, not identity or authorization. RFC 9309 defines user-agent groups and allow/disallow matching, while also establishing the protocol’s crawl-control role. Measure before and after: request volume, error rate, origin latency, and whether a legitimate Semrush project still completes.

How do robots.txt rules, responses, and propagation control Semrush?

Create a separate group for the exact documented user agent you intend to control, keep ordinary public content available, and publish the file at the root of every affected subdomain. Semrush states that each subdomain needs its own reachable /robots.txt; a rule on www does not control shop, docs, or another host.

Block only a costly route for the backlink crawler:

User-agent: SemrushBot
Disallow: /internal-search/

Block a specific audit crawler while leaving other agents untouched:

User-agent: SiteAuditBot
Disallow: /

Throttle the backlink crawler without naming Googlebot or AI search bots:

User-agent: SemrushBot
Crawl-delay: 5
Disallow: /cart/

Semrush documents wildcard support and says backlink SemrushBot honors Crawl-delay up to 10 seconds. Its crawler guidance also says a 4xx robots response means missing with no restrictions, 5xx stops crawling, and 3xx redirects are followed. Apply these details only as documented, then test the exact agent and host.

Rule changes may take up to an hour or roughly 100 requests to propagate for the backlink crawler. Fetch every host’s final 200 response, inspect its body, wait through one propagation window, then compare logs. Do not use 5xx as a permanent blocking policy.

Why can SiteAuditBot appear to be Google or OpenAI Search?

A Semrush Site Audit owner can configure crawler user-agent emulation, including Semrush, Google, or OpenAI Search options. That setting changes the presented audit identity; it does not turn SiteAuditBot into the real Google or OpenAI crawler. Diagnose it by correlating logs with the audit project, selected settings, timing, paths, and rate.

The Site Audit configuration guide documents user-agent choices and configurable crawl speeds, including a one-URL-per-two-seconds option where available. Treat that as project configuration evidence, not proof that any similarly named request came from Google or OpenAI.

Google and OpenAI publish separate controls. OpenAI’s official bot documentation distinguishes OAI-SearchBot from GPTBot. Allowing or blocking a Semrush agent does not directly change either role, and it does not change Google crawling. Keep those policies in separate, explicit groups.

How do you verify the fix without weakening security or SEO?

Verify a crawler-control change in 15 minutes by checking the published file, generating no destructive traffic, and comparing fresh logs with the baseline. Robots is public advice, not a security barrier. Keep sensitive routes behind authentication, use POST plus authorization for state changes, and never place secrets or private parameters in crawlable GET URLs.

Use this focused plan:

1. Minutes 0–3: fetch /robots.txt on every affected subdomain; record final URL, status, and rule.
2. Minutes 3–6: validate exact group spelling and confirm no Googlebot, OAI-SearchBot, or normal content rule changed.
3. Minutes 6–10: inspect logs for agent, path, rate, status, and cache behavior.
4. Minutes 10–15: compare the baseline, note propagation limits, and test the intended Semrush project.

Robots blocking is not reliable deindexing: Google explains that a blocked URL can still appear when other pages link to it in its robots.txt introduction. Use appropriate indexing controls on accessible pages and authentication for private data.

StoreCited can flag public crawler and AI-readiness controls visible at scan time. It does not live-monitor every bot request, prove the source of a request, force an AI citation, or replace access logs, WAF evidence, and authenticated security controls.