Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
-
Hi,
In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following:
Original (valid) URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green
Duplicate URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++**
These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem.
Thanks,
George
-
So glad to help, George!
-
Hi Chiaryn,
Thanks - you've been really helpful! I had assumed that as the referrer wasn't in the Web UI (per WMT), it wasn't available anywhere. I'd also assumed it was a copywriting issue and not a product data issue.
Need to readdress my assumptions
George
-
Hey George,
Thanks for writing in.
I looked into the pages with the ++ in the URL and it seems that they do actually exist on the site, so it isn't an issue with our crawler that is causing these in your crawl errors. For example, a link to the URL http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx?filter_colour=Green++ can be found in the source code of the page http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx here: http://screencast.com/t/HpHTlSs5gH8H
You can find the referral pages for the ++ pages on the site by downloading the Full Crawl Diagnostics CSV. In the first column, perform a search for the ++. When you find the correct row, look in the column labeled referrer, AM. This tells you the referral URL of the page where our crawlers first found the URLs that include ++. You can then visit this URL to find the links to those pages.
Since these URLs with the ++ do resolve with a 200 http status and they have the same code and content as the pages without the ++, our crawler will count them as duplicate content. I'm not certain why Screaming Frog and GWT are not find or reporting these pages; it may be that they parse the + signs in the URL differently than our crawler does.
As Keri and bishop23 mentioned, this is most likely not a major issue if GWT isn't reporting the errors, but we prefer to report the issues because we would rather be safe than sorry.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
-
I'm not seeing an answer that jumps out at me for this one. For the immediate future, don't sweat it if you're not seeing it in GWT. This is assigned to our help desk, and we'll have someone from there investigate more and get back to you, though it might be a few days because of the Thanksgiving holiday (if you don't get an answer today, it may be Monday before we have a chance to respond).
-
If they're not appearing on WMT than you should ignore unless it's an exact duplicated content, then delete
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved i got charged by the end of the trial against my will...
i am not using the tools , and I'm not interested in them in the future... how can i get a refund ...
Product Support | | alexx12120 -
I have assigned a Moz Pro seat to someone but they can't access a campaign's rankings report?
Hi there, I'm the account admin on a Moz Pro account. I have assigned a seat to a freelancer who's doing some work for me and they say they can't access a campaign's rankings report? The manage user's section shows them as having a seat license? It's been issued and accepted by them? Any ideas here? Thanks
Product Support | | NineFiftyNine.net0 -
Moz cant crawl site?
We're getting an error saying Moz is getting an errors crawling our client's site, but when I've put this though Google Search Console I'm not seeing any issues - any suggestions?
Product Support | | Ramarketingrob0 -
Rogerbot not crawling our site
Has anyone else had issues with Roger crawling your site in the last few weeks? It shows only 2 pages crawled. I was able to crawl the site using Screaming Frog with no problem and we are not specifically blocking Roger via robots.txt or any other method. Has anyone encountered this issue? Any suggestions?
Product Support | | cckapow0 -
Duplicated content issue?
Hi, Moz tools tell me that I have approx. 15xx Pages with High Priority Issues [Duplicate Page Content] going thru the details here are the majority: many products no longer exist and visitor redirected (404) to a same not found page. duplicated content with url appended with utm_source/utm_medium/utm_campaign GA tracking paras. On those page there are already canonical tags plus I've configed those parameters in Google Webmaster "URL Parameters" how much of these warning could compromise the seo proformance or provided that there have been dealt with should I just ignore these warning?
Product Support | | LauraHT0 -
Why can't the dates be changed in automatic reports?
I want to be able to change the publish date of automatic reports in my campaigns. One such campaign, which is a client campaign, it's set to run on the 8th after I selected "monthly". However, this doesn't work for me as this client want's to meet each month on between the 2nd and 5th of each month and I have to have this report data. So, I need to run this report on the first. Not the 2nd, not every 4 weeks... on the first. It seems like you guys have a fundamental flaw in the design of this tool, as great as it is. You've set the projects to auto run each month from the date it was added (at least from what I can tell). Provided that's true, then this would explain why the monthly reports won't work on my schedule because their on a weekly schedule instead. We, as clients, should be given the option to schedule when our scanning runs, when our reports get generated, etc. Every company runs their SEO and marketing differently, but the away you've set this up with the lack of options for us, forces us to work around your tools scheduling and not the other way around. Also, out of all of the SEO tools I've tried (quite a few), none have had this limitation. This should be addressed immediately. Thanks, Micha
Product Support | | Multiverse-Media-Group1 -
Where are my October Monthly Reports?
My custom monthly reports have not been generated for any of our clients for the month of October. The report lists 10/1/2013 as the last report and says that the next report will be available on 12/3/2013. Obviously, there is a glitch that has caused October to be skipped. How can I fix this in order to get my monthly reports to my clients ASAP?
Product Support | | accpar0 -
Duplicate Page and Title Content
We applied the tag to our website but Moz still picked up an error in Duplicate Page Content and Duplicate Title for http://www.corpsyn.com/ and http://www.corpsyn.com/index.html after the crawl was completed yesterday. From the forum last week, we tried using the rel=”canonical” tag to tell the Moz crawler that the two pages are the same on purpose but it didn't work. Below is how the code looks on our site up until the rel=”canonical” tag is added: http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
Product Support | | Corporate_Synergies
http://www.w3.org/1999/xhtml"> <title>Group Employee Health Benefits | National Insurance Broker</title>
css/style.css" type="text/css" />
css/style_index.css" type="text/css" />
”http://www.corpsyn.com” /> Is there something else we can try? Thanks0