Moz Crawler suddenly reporting 1000s of duplicates (BE.net)
-
In the last 3-4 days we've had several thousand 'duplicate content' warnings appear in our crawl report, 99% of them related to our on-site blog. The blog is BlogEngine.Net, but the pages simply don't exist. The majority seem to be Roger trying quasi-random URLs like:
/?page=410/?page=151
Etc. etc. The blog will present content for these requests, but it is of course the same empty page since there's only unique content for up to /?Page=10 or so.
Two questions:
1. Did something change recently? These blogs have been up for months, and this problem has only come up this week. Did Roger change to become more aggressive lately?
2. Suggested remediation? On one of the blogs I've put no-index no-follow for any page that has a /?page querystring, and we'll see what effect that has come next crawl next week. However, I'm not sure this will work as per:
http://moz.com/community/q/functionality-of-seomoz-crawl-page-reports
Anyone else had dynamic blogs suddenly blossom into thousands of duplicate content warnings? Google (rightly) ignores these pages completely.
-
Hate to bump my own question, but it appears I spoke too soon about no-index,no-follow solving this. The duplicate errors went away for about 5 days, but then yesterday spiked with the same problem. I've confirmed that no-index, no-follow are present on the pages being detected as bad.
As per the best practices document:
http://moz.com/learn/seo/robotstxt
Using meta robots no index no follow is the recommended option:
Block with Meta NoIndex
This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method
But it apparently isn't working, as evidenced by the new surge of duplicate errors. Is there anything else I can do? I don't want to explicitly block Roger in robots.txt as that seems rather backward. Should Roger be included the Bad Robots List?
-
Peter -
Thanks for the clarification. I understand the philosophy at hand, and I kind of even understood it before I had asked the question. I'm handling these with a mix of canonical and no-index/no-robot.
Related to that, update:
By marking the superfluous pages no-index/no-follow the error count for the site has diminished by about 10,000 and the warning count by about 28,000 so that seems to be the way to go. The pages that had content are 'low value' in this context, since that content was readily available elsewhere.
-
Hi there!
Thanks for writing in with a great question.
We definitely count those dynamic URLs as duplicate content. While we are pretty sure that search engines can figure this stuff out and know which URL to index, it's still considered best practices to canonicalize or otherwise direct crawlers to the original URL (as far as I know. I'm not a professional SEO so you might be better off asking the Pro Q&A community at www.moz.com/community/q - they are all SEOs like you).
Since some dynamic URL generators can cause problems for crawlers, we do try to be overly-inclusive of these issues rather than overly-exclusive. We want people to know about potential issues with sites, even if they're not really issues in the scheme of the site owner's specific SEO implementation plan.
In sum, we'd rather leave those judgments up to you and at the same time, provide you with the data you need to make these decisions. I hope this helps explain our thinking here! However, if you think that our crawler might be having issues, and you do not want to post your site urls here you could always send us a support ticket at help@moz.com. That way can can examine it a bit further and provide some insights into why our crawler thinks this way!
Hope this helps!
Peter
Moz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query on google analytic event report and customized report - Please help.
For my site I placed Black Friday offer.Now I have highlighted one product at homepage banner and tracking that with event. I am sharing with you actual figures of 1 day from Google analytic. Your proper response on this query will really really really helpful to me for understanding google analytic deeply and properly. So pls give your precious time for answering me, I will be thankful to you from bottom of my heart. I am checking event report from behavior, following is report for by that product event - Sessions - 30
Reporting & Analytics | | pragnesh9639
Unique Events - 30
Total Events - 34
Ecommerce Conversion Rate - 3.33% I am checking now report from All pages via Behaviour section - Page views - 97, Unique Page Views - 62, Entrances - 7, Bounce Rate - 57.14%, %Exit - 17.53% Now I create customize report for the same by adding
session -7
Ecommerce conversion rate - 0
unique events -4 MY query - when I added session in customized report then it is different from Event session why (7 and 30) ? same way Ecommerce conversion rate in customized report showing 0% why ( 0% and 3.33% ) ? same way unique events 4 and 30 ? do you suggest what is the best way to analysis such page or what will be best customize report for such analysis? Thanks0 -
Google Analytic - Avg load Time - Page Timing Sometimes Graph goes suddenly up why?
Hello All, I have attached the avg load time screenshot for my ecommerce site for 1 month. Screenshot is for Mobile site. If you check the graph then only one day graph gone very high. Now again I have attached that particular day graph too. My query is why graph gone high 1) Is it my site not performed well on particular one day on particular device for single visitor? because for desktop and tablet graph showing normal 2) so here if site restart or down then it can be a problem for desktop, tablet too right? Can anyone give me any clue? Z4hX1 IanuP
Reporting & Analytics | | pragnesh96390 -
My domain as a sufix on GA reports
Hi, There's a friend of mine who asked me with some weird results on her sites' Google Analytics reports (http://esther-roche.es). I've searched the GA code and its GA account config and I've no clue about where this esther-roche.es is to remove it. Any idea? Thanks in advance, BzgPcAo.jpg
Reporting & Analytics | | Webicultors0 -
Best way to handle duplicate title on Home page?
Moz reports two links to the same Home page ad duplicate titles ... http://myhjhome.com/index.php
Reporting & Analytics | | ElykInnovation
http://myhjhome.com I'm not sure if I should just 301 redirect http://myhjhome.com/index.php to http://myhjhome.com, or if there is a better way to handle that? Or should I comb the website and make sure all links to the Home page dont include index.php? Just looking for some extra help here, learning as I'm going, thanks!!0 -
Is the meta description available on the On Page Optimization Report even if its currently being optimized?
Currently, description is only available if the element is not being optimized (i.e. character count is off/keyword isn't included in the description)
Reporting & Analytics | | Jerome670 -
Moz Rank & Trust | Page vs Sub vs Root
Hey guys, Just need some help deciphering my OSE link metrics for my site theskimonster.com . Page MozRank: 5.51 (highest among my competitors) Page MozTrust: 5.74 (#2 among my competitors) Subdomain MozRank: 4.19 (#4 among my competitors) Subdomain MozTrust: 4.63 (#2 among my competitors) Root Domain MozRank: 3.89 (#5 or last place among competitors) Root Domain MozRank: 4.1 (#5 or last place among competitors) What does this mean? What am I doing right, what do I need to do?
Reporting & Analytics | | Theskimonster1 -
Duplicate Content
I am looking to check the duplicate content of two websites against each other, www.housesalesbulgaria.com and www.housesalesturkey.com. What is the best way to check this?
Reporting & Analytics | | Feily0 -
Campaign tracking and duplicate content
Hi all, When you set up campaign tracking in Google Analytics you get something like this "?variable=value parameters" in the URL. If you place such a link on your site as an internal link, will it be considered as a different URL and will have its own link value? The question I have is, since Google knows it's a Google link and knows the original URL (by stripping the tags), does it pass link value to the original URL? If not, what can be done to pass link value? Thanks in advance. Henry
Reporting & Analytics | | hnydnn0