Skip to content

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  • Unsolved

    roger-bot crawl error

    Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
    Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
    Ben Cloudflare activity log, showing differences in user agent strings
    2022-07-01_13-05-59.png

    Moz Pro | | BB_NPG
    0
  • Unsolved

    robots roger-bot

    Anyone got any ideas how to get a server/hosting provider who is preventing rogerbot from crawling and me not been able to set up a campaign to duplicate the error on there end? The server/hosting provider is crazydomains dot com My clients robots.txt User-agent: *
    Disallow: /wp-admin/
    Allow: /wp-admin/admin-ajax.php
    User-agent: rogerbot Disallow: Sitemap: https://www. something

    Moz Tools | | Moving-Web-SEO-Auckland
    0

Looks like your connection to Moz was lost, please wait while we try to reconnect.