Unsolved Replicate rogerbot error for server/hosting provider
-
Anyone got any ideas how to get a server/hosting provider who is preventing rogerbot from crawling and me not been able to set up a campaign to duplicate the error on there end?
The server/hosting provider is crazydomains dot com
My clients robots.txt
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent: rogerbotDisallow:
Sitemap: https://www. something
-
@tom-capper
Thanks Tom was more about server settings outside of robots.txt for hosting providers.
Managed to get them to remove rogerbot blocking on ALL there servers as they first stated that it was well known as a malicious robot hence the block???
Until I made them aware.... cough cough -
There are a bunch of third party tools that would allow you to validate robots.txt against a given user agent.
For example:
- https://technicalseo.com/tools/robots-txt/ (includes Moz user agents in dropdown)
- https://www.realrobotstxt.com/ (allows you to copy/paste a test robots file)
Without the hosting provider having a Moz account & campaign of their own, this is probably the easiest way to get them to replicate the error.
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved How I can update my new campaign in Moz?
I am facing a problem when I solve any issue in my running campaign.
Moz Tools | | ClippingOutsources
Is there any option to see once I fix the issue and my campaign shows it instantly or how I can see that my exiting issue after I fix it? Can any one suggest me, please?0 -
Unsolved Landing pages report has no data even if I have ranking keywords and traffic
Is there any reason my landing page report does not include data for pages? I'm sure there is organic traffic on them, and I have tracked the correct keywords. Any similar insight will be helpful.
Moz Tools | | davidevans_seo0 -
Unsolved How to check the DA history?
Hello everyone!
Moz Tools | | ZeonAley
I want to know how to check the DA history of my website Inkjet Wholesale. Because I don't have any idea about Moz that how to use it. If anyone knows about DA's history check on Moz, please let me know.0 -
Unsolved Rogerbot blocked by cloudflare and not display full user agent string.
Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
Moz Pro | | BB_NPG
Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
Ben Cloudflare activity log, showing differences in user agent strings
2022-07-01_13-05-59.png0 -
Unsolved Domain Authority Not Changed?
The domain authority has not changed for a few months; even I got some high-quality links from authority sites, and Moz has not crawled them yet? Can someone tell me how much time it generally Moz takes to crawl new links for a domain and update DA?
Moz Tools | | peaceforchange00
My domain is https://thebookspdf.net/2 -
Unsolved domain authority has not changed
Hello, can you help me? The domain authority has not changed for months, and in fact, the ranking has come down. But my links have increased a lot but have remained the same
Moz Tools | | semrtaz
this is my domain 😧 https://nezafatnews.ir/1 -
Unsolved error in crawling
hello moz . my site is papion shopping but when i start to add it an error appears that it cant gather any data in moz!! what can i do>???
Moz Tools | | valigholami13860
