Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Unsolved Rogerbot blocked by cloudflare and not display full user agent string.

Moz Pro

1114

BB_NPG Subscriber last edited by

Hi,

We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help.

robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it.

When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ?

Thanks
Ben

Cloudflare activity log, showing differences in user agent strings
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved Moz crawler not crawling on my site

crawl error robots.txt

Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh

0
Unsolved Strange "?offset" URL found with content crawl issues

crawl error url issue

I recently recieved a slew of content crawl issues via Moz for URL's that I have never seen before For example:
Standard URL: https://skilldirector.com/news,
Newly identified URL: https://skilldirector.com/news?offset=1469542207800&category=Competency+Management). Does anyone know where the URL comes from and how to fix it?
Moz Pro | | HannahPalamara

0
Unsolved Moz can't crawl my site

crawl error crawl errors

Moz is being blocked from crawling the following site - https://www.cleanchain.com. When looking at Robot.txt, the following is disallowing access but don't know whether this is preventing Moz from crawling too? User-agent: *
Disallow: /adeci/
Disallow: /core/
Disallow: /connectors/
Disallow: /assets/components/ Could something else be preventing the crawl?
Moz Pro | | danhart2020

0
Its been over a month, rogerbot hasn't crawled the entire website yet. Any ideas?

Rogerbot has stopped crawling the website at 308 pages past week and has not crawled the website with over 1000+ pages. Any ideas on what I can do to get this fixed & crawling?
Moz Pro | | TejaswiNaidu

0
How do I get our full domain link profile?

With Open Site Explorer I have exported our domain link profile but I only get 65 domains returned when our marketing software reports that we have 315 . . . I know that some domains point to www.bluelink.ca which redirects to www.bluelinkerp.com. Is Open Site Explorer missing these links?
Moz Pro | | BlueLinkERP

0
Is there a way to see what keywords users of my site are using to find it online?

Since Google Analytics no longer shows the keywords used by people to find a site online, does the SEOMoz toolset provide somethng to show this data?
Moz Pro | | Mionkeybot

0
Can we assign multiple admins and users with different role in one PRO account?

We wish to give different login details to authorized personnel without handing out the main account ownership login.
Moz Pro | | solution.advisor

0
How to get rid of the message "Search Engine blocked by robots.txt"

During the Crawl Diagnostics of my website,I got a message Search Engine blocked by robots.txt under Most common errors & warnings.Please let me know the procedure by which the SEOmoz PRO Crawler can completely crawl my website?Awaiting your reply at the earliest. Regards, Prashakth Kamath
Moz Pro | | 1prashakth

0