Crawler Warnings
What's Covered?
This guide offers more information about the types of issues flagged in the Crawler Warnings section of Site Crawl.
Quick Links
Moz Pro Site Crawl Crawler Warnings
Meta Noindex
This page contains a noindex value in the meta robots tag, which directs search engines to exclude this page from their indexes. This is different than a nofollow tag which we’ll explain in the next section. Noindex pages will still have their links followed by crawlers, they just won’t be included in indexes. Depending on your page or circumstance, this may not be an issue that needs fixing.
If you determine that this page should be indexed by search engines, inspect the <head> of your page and change the following to say index rather than noindex.
<p><META NAME="robots" CONTENT="noindex"></p>
Or simply delete the meta robots tag completely. Assuming this page isn't blocked from search engines in some other fashion (in and X-Robots header or in the robots.txt file), search engines should start indexing this page again.
If you determine that these pages are correctly tagged and you do not want them indexed, you can ignore these issues in your Site Crawl so that they do not continue to be flagged with each crawl.
X-Robots Nofollow
Search engines have been directed not to follow any of the links on this page due to a value of nofollow included in the x-robots response header.
This tells search engines not to crawl the links on your page. This becomes a problem when your pages accumulate links that could be passing along link equity (ranking power) to other pages on your site, but are not able to because the links are never crawled. Depending on your page or circumstance, this may not be an issue that needs fixing. If you determine that you don’t want the links on these pages followed, you can ignore these issues in your Site Crawl so that they do not continue to be flagged with each crawl.
This can also appear as: X-Robots-Tag: googlebot: nofollow OR X-Robots-Tag: otherbot: index, nofollow.
Meta Nofollow
This tells search engines not to crawl the links on your page. This becomes a problem when your pages accumulate links that could be passing along ranking power to other pages on your site, but are not able to since the links are never crawled. Depending on your page or circumstance, this may not be an issue that needs fixing.
If you determine that the links on this page should be followed, inspect the source code of your page and change to say follow, or simply delete that element altogether. Assuming this page isn't set to nofollow in some other fashion (such as in the X-Robots header), search engines should start crawling links on this page again.
X-Robots Noindex
This tells search engines not to index your page, which prevents your page from being found in search engines. Depending on your page or circumstance, this may not be an issue that needs fixing.
If you determine that the links on this page should be indexed, remove X-Robots-Tag: noindex from your HTTP header. This can also appear as: X-Robots-Tag: googlebot: noindex or X-Robots-Tag: otherbot: noindex, nofollow.
Tracking Issue Resolution Progress
As you work through issues flagged in your Campaign’s Site Crawl section you can track your progress by marking issues as fixed or by ignoring them.
Mark as fixed: Marking an issue as Fixed will indicate to our crawler that you have taken action on your site to resolve the issue. During the next crawl of your site, the issue will be removed from the interface if the crawler does not encounter the same issue again. If the issue is still found it will be marked as Not fixed in your Site Crawl data.
Ignore: Marking an issue as Ignore will indicate to our crawler that you are aware of the issue but are not planning to take action to resolve it. The crawler will not include the issue in upcoming crawls as a result. You can Unignore issues at any time from your Campaign Settings.
To mark issues as Fixed or Ignore, select them using the checkboxes on the left and then clicking Mark as fixed or Ignore from the top right.
For issues marked as Fixed, if our crawler does not encounter those issues in the next crawl they will be removed from your list of site crawl issues. If the issue is still found, it will be marked as Not fixed.
For issues marked as Ignore, our crawler will not report those issues in your next crawl. They will be removed from your list of site crawl issues. You can Unignore issues at any time.
Related Articles
Was this article helpful?
Yes! Amazing!
Yes! It was what I needed.
Meh. It wasn’t really what I was looking for.
No, it wasn’t helpful at all.
Thanks for the feedback.