Crawl Diagnostics Error Spike
-
With the last crawl update to one of my sites there was a huge spike in errors reported. The errors jumped by 16,659 -- majority of which are under the duplicate title and duplicate content category.
When I look at the specific issues it seems that the crawler is crawling a ton of blank pages on the sites blog through pagination.
The odd thing is that the site has not been updated in a while and prior to this crawl on Jun 4th there were no reports of these blank pages.
Is this something that can be an error on the crawler side of things?
Any suggestions on next steps would be greatly appreciated. I'm adding an image of the error spike
-
This would be another issue. I would need to look at the code to give you more insight. But off the bat I assume that this is an issue regarding mislabeling the rel=next and rel=prev. They can be kind of tricky to work with on a broad based update due to the fact that they are intended to refer to specific pages. If you do not have the end page labeled Google says :
"When implemented incorrectly, such as omitting an expected rel="prev" or rel="next" designation in the series, we'll continue to index the page(s), and rely on our own heuristics to understand your content."
I would look into this first. If the answer is still elusive to you the next option would probably be finding a different set of eyes on the code to see if there are any minor oversights that you may have overlooked.
-
One last thing;
It seems that I have a game plan for addressing this issue, but as I think about this one thing has me concerned in the way Roger crawled the site.
The site has maybe a total of 100 articles, which would account for ?Page=10, but what I'm seeing is errors on ?Page=104. When you look at that page its a blank. Where is Roger coming up with that parameter?
Do you think this is a Roger issue or something else?
-
Makes sense
-
Unless you have some super secret page that is buried somewhere deep down in your site that you can ONLY get to from those pages, it wouldn't make sense to have them follow the links. All that will happen is they land on the next page, scrape it to the noindex tag and move on. They won't index and this just waste your sites bandwidth and slows everything else down. If it's a noindex it should usually be a nofollow unless you are looking to track conversions or some other specific only navigable through those pages.
-
Hey Jake;
Whats your option of using "nofollow" vs "follow" on the pages i'm blocking from indexing? Is there a reason to prevent them from following the links on these pages?
-
Cool glad we could help!
if you want to clean up your code and are posting site wide for them I would recommend the none tag
Accounts for both
noindex, nofollow
-
Thank you again for the input, the goal here is not provide accurate reporting and ensure that the site conforms to the search engines requirements.
Currently the "?page=" parameter is not blocked through . it sounds like this maybe the issue.
I will update the code to address that and see what kind of results we get with the next update. I think this is best addressed at the code level, rather then the robots.txt.
Thanks
-
Rodger crawls like the Google bot and takes his hints from the robot.txt file. So whatever Rodger is seeing is usually what the other spiders are seeing as well. From time to time I have encountered slight glitches to the SEOmoz crawler as they change and update their algorithm.
When it comes down to it, Google examines a link profile through a microscope akin to the Large Hadron Collider. where as we have to examine it through a magnifying glass from 1935.
The wonderful people here at SEOmoz are always trying to give us a better view, but it is still imperfect. I would say if all else fails and this report continues to show errors in moz then get your reports for your clients directly from webmaster tools.
-
** How do I tell Roger no to crawl these blank pages?**
Any easy solution is to block roger in robots.txt
User-agent: rogerbot
Disallow: [enter pages you do not wish to be crawled]
But a better solution would be to fix the root problem. If your only goal is to provide clean reporting to your client the above will work. If your goal is to ensure your site is crawled correctly by Google/Bing, then Jake's suggestion will work. You can help Google and Bing understand your site by telling them how to handle parameters.
I would prefer to fix the root issue though. Do the pages which are being reported as duplicate content have the "noindex" tag on them? If so, you can report the issue to the moz help desk (help@seomoz.org) so they can investigate the problem.
-
Hey Jake;
Thanks for your feedback, i did make some changes to the code (posted in the reply to Jamie). I'll take a closer look at the webmaster tools to make sure things are OK on that end.
FYI: The "rel=prev / rel=next tags" are implemented
I added code to manage
to pages that are accessed through
- /Blog/?tag=
- /Blog/category/
- /Blog/archive.aspx
As a secondary concern, with Roger now reporting all these issues in SEOMoz, I provide these reports to my clients and thus having 16k errors is not a good PR thing. How do I tell Roger no to crawl these blank pages?
-
It looks like Rodger found his way into your variable URLs!
This could definitely cause a problem if the engine crawlers are seeing this path as well. Have you made any changes to the code on your site or the URL structure lately?
Regardless, you might want to examine in your Webmaster Tools for both Google and Bing.
For Google you will want to check the blocked URL's under the Health menu. This will give you the information on what pages are and are not blocked. If you notice that the Head Match term you are looking to exclude is not listed make sure that you upload the term to the robots.txt file on your site. Other fixes for this include canonicalisation tagging or the implementation of the rel=prev / rel=next tags. There are a few other ways that are more complicated and I recommend avoiding unless absolutely necessary.
But good news everyone! Google has a few ways to go about fixing the indexation.
Bing is a little Different but just as easy. In the Bing Webmaster Tools under the Index tab, there is a tool called URL Nor<a class="cpad Subject message-low-priority-icon marginleft5 bold">malization</a> you can tell the crawlers to exclude a portion of the query string without changing anything on your database. It also automatically finds and suggests <a class="cpad Subject message-low-priority-icon marginleft5 bold">query parameters for normalization as well. This is a recent change for Bing and could account for the sudden jump in warnings.</a>
I hope this helps and you keep being awesome!
-
Hey Jamie;
In an effort to block crawling of pages on the blog that are essentially duplicating content I added coded (on (4/16) to insert :
to pages that are accessed through
/Blog/?tag=
/Blog/category/
/Blog/archive.aspx
I did not do this for
/Blog/?page=
There were no changes to the robots.txt
There were no updates to canonical tag
There were no updates to pagination
Thanks for your prompt reply
-
Can you share what changes have been made to the site? A few ways this can happen are:
-
a change to the robots.txt file
-
a change to your site's template either removing a canonical tag, a noindex tag, or altering your pagination in any way such as modifying paginated titles
-
resolving an onsite issue which prevented crawling of these pages
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Using Weglot on wordpress (errors)
Good day to you all, Does anyone have experience of the errors being pulled up by Moz about the utility of the weglot plugin on Wordpress? Moz is pulling up URLs such as: https://www.ibizacc.com/es/chapparal-2/?wg-choose-original=false These are classified under "redirect issues" and 99% of the pages are with the ?wg-choose parameter in the URL. Is this having an actual negative impact on my search or is it something more Moz related being highlighted. Any advice be appreciated and a resolution .. Im thinking I could exclude this parameter.
Moz Pro | | alwaysbeseen0 -
How do fix an 803 Error?
I got am 803 error this week on the Moz crawl for one of my pages. The page loads normally in the browser. We use cloudflare. Is there anything that I should do or do I wait a week and hope it disappears? 803 Incomplete HTTP response received Your site closed its TCP connection to our crawler before our crawler could read a complete HTTP response. This typically occurs when misconfigured back-end software responds with a status line and headers but immediately closes the connection without sending any response data.
Moz Pro | | Zippy-Bungle1 -
Is it possible to block Moz from crawling sites?
Hi, is it possible to stop Moz from crawling a site at the server level? Not that I am looking to do this or anything, but here's why I'm asking. I have been crawling a site that is managed (currently by 2 parties), and I noticed that this week pages crawled went from 80 (last week) to 1 page!! I know, what? See my image attached... and the issues all went to zero "0"....! So is it possible that someone can't prevent Moz from crawling the site at the server level? I checked the robots.txt file on the site, but nothing there. I'm curious. dYNUwjd.jpg
Moz Pro | | co.mc0 -
Crawl Diagnostics: Next crawl date is in the past
Hi - I have quite a few crawl diagnostic errors and warnings. I have attempted to fix many of them but noticed this note at the bottom of the crawl diagnostics chart: "Last Crawl Completed: Mar. 22nd, 2013 Next Crawl Starts: Mar. 29th, 2013" It looks like SEOMoz thinks the next crawl date is Mar 29th, 2013, which is two weeks ago. Is there any way to "force" the crawl and get it back on regular schedule? This may have happened when my account was disabled because my credit card expired...Thoughts?
Moz Pro | | 6thirty0 -
Crawl Test - Taking too long
The last crawl test I invoked seems to be in progress for over 24 hours. The one before that completed in a few hours. Wish there was a progress indicator or an option to cancel. The crawl (from Tool > Crawl Test) should not take this long. Any ideas or suggestions? Also, the keyword research tool (plus a few others) have been down ever since I signed up. Is this a normal?
Moz Pro | | MomoMasta0 -
Help with URL parameters in the SEOmoz crawl diagnostics Error report
The crawl diagnostics error report is showing tons of duplicate page titles for my pages that have filtering parameters. These parameters are blocked inside Google and Bing webmaster tools. I do I block them within the SEOmoz crawl diagnostics report?
Moz Pro | | SunshineNYC0 -
I have a Rel Canonical "notice" in my Crawl Diagnostics report. I'm presuming that means that the spider has detected a rel canonical tag and it is working as opposed to warning about an issue, is this correct?
I know this seems like a really dumb question but the site I'm working on is a BigCommerce one and I've been concerned about canonicalisation issues prior to receiving this report (I'm a SEOmoz pro newbie also!) and I just want to be clear I am reading this notice correctly. I presume this means that the site crawl has detected the rel canonical tag on these pages and it is working correctly. Is this correct?? Any input is much appreciated. Thanks
Moz Pro | | seanpearse0 -
Crawl Diagnostics bringing 20k+ errors as duplicate content due to session ids
Signed up to the trial version of Seomoz today just to check it out as I have decided I'm going to do my own SEO rather than outsource it (been let down a few times!). So far I like the look of things and have a feeling I am going to learn a lot and get results. However I have just stumbled on something. After Seomoz dones it's crawl diagnostics run on the site (www.deviltronics.com) it is showing 20,000+ plus errors. From what I can see almost 99% of this is being picked up as erros for duplicate content due to session id's, so i am not sure what to do! I have done a "site:www.deviltronics.com" on google and this certainly doesn't pick up the session id's/duplicate content. So could this just be an issue with the Seomoz bot. If so how can I get Seomoz to ignore these on the crawl? Can I get my developer to add some code somewhere. Help will be much appreciated. Asif
Moz Pro | | blagger0