A question about Mozbot and a recent crawl on our website.
-
Hi All,
Rogerbot has been reporting errors on our website's for over a year now, and we correct the issues as soon as they are reported.
However I have 2 questions regarding the recent crawl report we got on the 8th.
1.) Pages with a "no-index" tag are being crawled by roger and are being reported as duplicate page content errors. I can ignore these as google doesnt see these pages, but surely roger should ignore pages with "no-index" instructions as well? Also, these errors wont go away in our campaign until Roger ignores the URL's.
2.) What bugs me most is that resource pages that have been around for about 6 months have only just been reported as being duplicate content. Our weekly crawls have never picked up these resources pages as being a problem, why now all of a sudden? (Makes me wonder how extensive each crawl is?)
Anyone else had a similar problem?
Regards
GREG
-
Its pretty big
Over 1000 Pages in the index, and many more internal URLs to crawl that have a no-index tag. (booking forms etc)
Ill see if we can archive our other campaigns and let roger crawl our main site properly.
-
How big is your website Greg ?
-
Thanks Nakul,
I do a weekly scan with Xenu which doesn't have a URL limit like SF.
I was under the impression a full scan of the site was done each week, but as you say, its being scanned in chunks, divided across our 3 other websites.
If this is the case, it would be great to let Mozbot know were to crawl to avoid unnecessary resources being used up when it could be scanning our most important pages.
Greg
-
Greg The crawl is limited to 10,000 (Total) for all your 5 campaigns. As far as whether or not Roger-Bot should ignore Noindex - Here's what I think - I think the intent of that tool here is to find issue. In this scenario, Roger bot is making sure you are aware of the fact that some of those pages have a noindex. Roger does not know whether it's intentional or not. You can also do a deeper crawl and do a deep dive into your website by using Screaming Frog SEO Spider http://www.screamingfrog.co.uk/seo-spider/ It does a great job of doing a deep crawl when you want it since it's a desktop software and you can set all sorts of options and identify issues.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Trailing forward dash question
This might be a very newbie technical SEO question, but here goes.This page, with no trailing forward slash:idearocketanimation.com/whiteboard-animation-oHas a Moz page authority of 45 while this page:
Moz Pro | | Wagster
idearocketanimation.com/whiteboard-animation-o/Has a Moz page authority of 1. Right now, the former is forwarding to the latter. Might Google not see the links I've built up to it, like Moz isn't? Can this be fixed in WordPress?I appreciate your responses.0 -
Good tool to track external links from the website
I am in search of a tool that provides me links generating from my site to another site. Is there a software or tool that can scan the whole site and provide me what are the links of other sites in my site.
Moz Pro | | csfarnsworth0 -
Website disappeared from Google :(
Dear all, I've a 5 years old website, which was in top rank for a particular keyword. It's been a couple of months that the site suddenly disappeared from Google.com.au search. However, the site is number one on Bing and Yahoo.
Moz Pro | | jarrodb
I've tried almost everything in my limited knowledge but it doesn't seem to come back. Can you please suggest if something can be done. Google.com.au
website: http://partysuppliesnow.com.au/
keyword: "party supplies" Your help would be highly appreciated.
Thanks in advance 🙂
Jarrod0 -
54 new 404 errors on my website?
Hi There In the latest report I have 54 404-errors. All last week, previously I had 2 404s that I fixed. In report say: Title404 : ErrorMeta DescriptionTraceback (most recent call last): File "build/bdist.linux- x86_64/egg/downpour/init.py", line 391, in _error failure.raiseException() File "/usr/local/lib/python2.7/site- packages/twisted/python/failure.py", line 370, in raiseException raise self.type, self.value, self.tb Error: 404 Not FoundMeta RobotsNot present/emptyMeta RefreshNot present/empty Are these normal 404 errors I have to look at and fix? Or is this some script that running on my server and causing errors? In general - what should I do to fix this? Thanks Dean
Moz Pro | | Passanger880 -
Crawl Errors Confusing Me
The SEOMoz crawl tool is telling me that I have a slew of crawl errors on the blog of one domain. All are related to the MSNbot. And related to trackbacks (which we do want to block, right?) and attachments (makes sense to block those, too) ... any idea why these are crawl issues with MSNbot and not Google? My robots.txt is here: http://www.wevegotthekeys.com/robots.txt. Thanks, MJ
Moz Pro | | mjtaylor0 -
Why won't scheduled crawl of my site begin?
I currently have a campaign running on SEOMoz for over a month. It has been showing that a crawl was scheduled to start on 12/21. Now it's 12/23 and there has not been a new crawl, and it still says scheduled for 12/21.. Anyone know why this is happening or how to fix it? Thanks
Moz Pro | | Prime850 -
Should I setup separate campaigns for my website and blog?
Both the website and blog are hosted on the same domain, just wondering if I should setup separate campaigns under SEOmoz for each? If so, why would this be helpful? http://www.tatango.com http://www.tatango.com/blog
Moz Pro | | Tatango0 -
Trying to understand how a website is getting higher than me with less links
Hi i am new to seo and trying to teach myself the best way to improve a site and the best way to use the tools on seomoz. The problem i have is. i am working on a page at the moment on a site. the page is called weight loss hypnotherapy http://www.clairehegarty.co.uk/weight-loss-hypnotherapy and i have around 130 links going to the page where as a site which is much higher than me in google has only around 5 links. I cannot understand with being new to seo how this can happen, can anyone please explain what i need to do to improve my ranking please. here is the site i am talking about that is higher than me www.weightlosshypnotherapy.co.uk/ any help would be great can anyone also give me a good example of a page before it has been optimised and a page after it has had this done.
Moz Pro | | ClaireH-1848860