Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Unsolved Moz crawler not working

Moz Pro

293

teikh Subscriber last edited by

Hi

Moz crawler keep failing on my site with the error showing : Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag.

I'm not sure what am I missing out.. this is my robots.txt.. i don't think Im missing anything else.. https://www.wearefutureheads.com/robots.txt

can the support team help ?
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Unsolved Moz crawler not crawling on my site

crawl error robots.txt

Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh

0
How get rid of 403 crawl error?

403 errors crawl error

My wordpress website has 162 crawl 403 errors. Based on what I read it means that the server is denying crawlers to access the pages. The pages itself will load so guessing it's just an issue with crawlers only. How do I go about fixing this issue?
On-Page Optimization | | emrekeserr3

0
Unsolved How do I cancel this crawl?

crawl error crawl in progress crawl stalled

The latest crawl on my site was the 4th Jan with a current crawl 'in progress'. How do i cancel this crawl and start a new one? I've been getting keyword ranking etc but no new issues are coming through. Screenshot 2022-05-31 083642.jpg
Moz Tools | | ClaireU

0
Unsolved /%25s

25 404 error crawl error

Hi Community, has anyone else had a 404 error reported by Moz, where the end of the domain is /%25s? The error comes from my blog home page https://kaydee.net/blog/ But when I look at the source code, I can't see anything that has a space at the end of the URL. I wonder if it is to do with the WordPress search? Thanks in advance for any insight.
Moz Pro | | kaydeeweb

0
Issues with Moz producing 404 Errors from sitemap.xml files recently.

My last campaign crawl produced over 4k 404 errors resulting from Moz not being able to read some of the URLs in our sitemap.xml file. This is the first time we've seen this error and we've been running campaigns for almost 2 months now -- no changes were made to the sitemap.xml file. The file isn't UTF-8 encoded, but rather Content-Type:text/xml; charset=iso-8859-1 (which is what Moveable Type uses). Just wondering if anyone has had a similar issue?
Moz Pro | | BriceSMG

0
Stupid question: why SEOMOz has become MOZ?

According to their On-Page Report Card the URL should include the main keyword i.e. SEO in their case, so why do they get an F grade If I check their main page www.moz.com for the keyword SEO? Thanks for your comments!
Moz Pro | | mcany

0
My OnPage dosent work, why?

When im trying to use the on-page grade analysis seomoz says that is empty and there is now keywords.. this happens with 3of 5 of my campaigns.. Please help
Moz Pro | | majako

0
SEOmoz crawler and duplicate content

Does anybody know if the SEOmoz crawler picks up canonical tags when its looking for duplicate content? I've got a ton of errors in one of my projects even though they all have canonical tags. Thanks!
Moz Pro | | neooptic

0