Moz Crawler suddenly reporting 1000s of duplicates (BE.net)
-
In the last 3-4 days we've had several thousand 'duplicate content' warnings appear in our crawl report, 99% of them related to our on-site blog. The blog is BlogEngine.Net, but the pages simply don't exist. The majority seem to be Roger trying quasi-random URLs like:
/?page=410/?page=151
Etc. etc. The blog will present content for these requests, but it is of course the same empty page since there's only unique content for up to /?Page=10 or so.
Two questions:
1. Did something change recently? These blogs have been up for months, and this problem has only come up this week. Did Roger change to become more aggressive lately?
2. Suggested remediation? On one of the blogs I've put no-index no-follow for any page that has a /?page querystring, and we'll see what effect that has come next crawl next week. However, I'm not sure this will work as per:
http://moz.com/community/q/functionality-of-seomoz-crawl-page-reports
Anyone else had dynamic blogs suddenly blossom into thousands of duplicate content warnings? Google (rightly) ignores these pages completely.
-
Hate to bump my own question, but it appears I spoke too soon about no-index,no-follow solving this. The duplicate errors went away for about 5 days, but then yesterday spiked with the same problem. I've confirmed that no-index, no-follow are present on the pages being detected as bad.
As per the best practices document:
http://moz.com/learn/seo/robotstxt
Using meta robots no index no follow is the recommended option:
Block with Meta NoIndex
This tells engines they can visit, but are not allowed to display the URL in results. This is the recommended method
But it apparently isn't working, as evidenced by the new surge of duplicate errors. Is there anything else I can do? I don't want to explicitly block Roger in robots.txt as that seems rather backward. Should Roger be included the Bad Robots List?
-
Peter -
Thanks for the clarification. I understand the philosophy at hand, and I kind of even understood it before I had asked the question. I'm handling these with a mix of canonical and no-index/no-robot.
Related to that, update:
By marking the superfluous pages no-index/no-follow the error count for the site has diminished by about 10,000 and the warning count by about 28,000 so that seems to be the way to go. The pages that had content are 'low value' in this context, since that content was readily available elsewhere.
-
Hi there!
Thanks for writing in with a great question.
We definitely count those dynamic URLs as duplicate content. While we are pretty sure that search engines can figure this stuff out and know which URL to index, it's still considered best practices to canonicalize or otherwise direct crawlers to the original URL (as far as I know. I'm not a professional SEO so you might be better off asking the Pro Q&A community at www.moz.com/community/q - they are all SEOs like you).
Since some dynamic URL generators can cause problems for crawlers, we do try to be overly-inclusive of these issues rather than overly-exclusive. We want people to know about potential issues with sites, even if they're not really issues in the scheme of the site owner's specific SEO implementation plan.
In sum, we'd rather leave those judgments up to you and at the same time, provide you with the data you need to make these decisions. I hope this helps explain our thinking here! However, if you think that our crawler might be having issues, and you do not want to post your site urls here you could always send us a support ticket at help@moz.com. That way can can examine it a bit further and provide some insights into why our crawler thinks this way!
Hope this helps!
Peter
Moz Help Team.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google analytics suddenly stopped tracking all my landing pages
Hey guys. I love the new update of GA. Looks so clean. So, of course, I was excited to see how my landing pages were doing. I went to behavior, all content, all pages. And I noticed it's only showing me 19 pages out of the 93 I have indexed. And none of the top ones at all! Can't find them anywhere in GA! Anyone seen this before? Thank you so much
Reporting & Analytics | | Meier0 -
Google Analytics Automated Reporting
HI all, I tend to do a big reporting powerpoint deck using screenshots from google analytics and tables I create year end and mid year. It's like an 80 page report for the 10 webisite swe have and then I go ahead and make annotations as I see from the data. That being said this can take a lot of time, up to a 40 hours of time to pull it all together or more which is challenging when you have daily meetings. Anyhow, I've looked into automating and tried a couple things: 1. Tableau- but it keeps crashing and seems tedious 2. Dashlane and supergrabber- seem a bit tedious to set up too. Anyone have ideas on how to better shar ereporting in the organization in this type of format for a website (websites)? Organic, paid, traffic, etc. Laura
Reporting & Analytics | | lauramrobinson322 -
Google Analytics Landing Page Report Discrepancy
I have noticed that when I run a landing page report and use the advanced option so I can view only the landing pages that include a particular string in the URL, have noticed that I in the report, the graph at the top will say one thing, but the data below says something else. For example, the graph for one particular search shows 200 Impressions, but the info below says 700 impressions and 610 clicks. Anyone seen anything similar or have any ideas why? Thanks! Craig
Reporting & Analytics | | TheCraig0 -
Google Analytics Report throws up Google as a referrer
Good morning from Wet & Windy 12 degrees C wetherby UK... Using Google analytics I've noticed in the traffic sources refferer subsection some traffic is categorized as originating from Google. Whats puzzling me is.... I know a huge amount of traddic stems from Google but as the below screenshot illustrates only 21 visitors come from Google: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-refferal-sources-top-levelcopy.jpg And when i drill down some are coming from Google mobile 😞 http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-referral-sourcescopy.jpg Is traffic categorised as Google referrer down to Google hiding searches via ssl as explained here: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/google-referral-sourcescopy.jpg Any insights welcome 🙂
Reporting & Analytics | | Nightwing0 -
Basic Purpose of SEO moZ s subscription !!!!!
How the subscription for seomoz will help to improve the traffic of my website ?
Reporting & Analytics | | fullerenedr0 -
Nofollow page is being reported as a landing page for organic search in Google Analytics
One of my client's websites includes a series of pages for an enrollment process. All of these pages are blocked by robots.txt. In Google Analytics these pages are showing data as landing pages for organic search traffic, and have been for quite some time. There was recently a surge of organic search traffic landing on one of these pages, coming from multiple search engines. The pages appear to be blocked and I'm not finding any of them in the search results for the keywords that are being reported in GA or by searching for the url. Does anyone have any insights into why this might be happening?
Reporting & Analytics | | rgibson1000 -
On what report do I get to know where do the external links come from?
I need to get a list of the external links. I don't find that report, where do I get it?
Reporting & Analytics | | carloscontinua0 -
URL Structure and Analytics Reporting
Since there are no dumb questions, here's mine: Our url structure currently displays the date followed by the post title. However, our news website has many categories like world, sports, entertainment etc. Now, when tracking in GA we get to see how many hits a specific page is getting (like entertainment) but since the category is not defined in the URL GA does not add the detail story views to the category page views. To illustrate further, a story from the entertainment section maybe displayed on the Home page and a reader may click on that story and read it without going to the entertainment page. This however, does not reflect in the analytics for that specific page(in this example, the entertainment page). My question is, should we ammend the url structure to include the categories, if so, would the editor have to do that while composing the post or is there some other way. We are using WP. Or should I create some filters or something on GA. Thanks!
Reporting & Analytics | | RishadShaikh590