Duplicate Content Report: Duplicate URLs being crawled with "++" at the end
-
Hi,
In our Moz report over the past few weeks I've noticed some duplicate URLs appearing like the following:
Original (valid) URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green
Duplicate URL:
http://www.paperstone.co.uk/cat_553-616_Office-Pins-Clips-and-Bands.aspx?filter_colour=Green**++**
These aren't appearing in Webmaster Tools, or in a Screaming Frog crawl of our site so I'm wondering if this is a bug with the Moz crawler? I realise that it could be resolved using a canonical reference, or performing a 301 from the duplicate to the canonical URL but I'd like to find out what's causing it and whether anyone else was experiencing the same problem.
Thanks,
George
-
So glad to help, George!
-
Hi Chiaryn,
Thanks - you've been really helpful! I had assumed that as the referrer wasn't in the Web UI (per WMT), it wasn't available anywhere. I'd also assumed it was a copywriting issue and not a product data issue.
Need to readdress my assumptions
George
-
Hey George,
Thanks for writing in.
I looked into the pages with the ++ in the URL and it seems that they do actually exist on the site, so it isn't an issue with our crawler that is causing these in your crawl errors. For example, a link to the URL http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx?filter_colour=Green++ can be found in the source code of the page http://www.paperstone.co.uk/cat_553_Desktop-Essentials.aspx here: http://screencast.com/t/HpHTlSs5gH8H
You can find the referral pages for the ++ pages on the site by downloading the Full Crawl Diagnostics CSV. In the first column, perform a search for the ++. When you find the correct row, look in the column labeled referrer, AM. This tells you the referral URL of the page where our crawlers first found the URLs that include ++. You can then visit this URL to find the links to those pages.
Since these URLs with the ++ do resolve with a 200 http status and they have the same code and content as the pages without the ++, our crawler will count them as duplicate content. I'm not certain why Screaming Frog and GWT are not find or reporting these pages; it may be that they parse the + signs in the URL differently than our crawler does.
As Keri and bishop23 mentioned, this is most likely not a major issue if GWT isn't reporting the errors, but we prefer to report the issues because we would rather be safe than sorry.
I hope this helps. Please let me know if you have any other questions.
Chiaryn
-
I'm not seeing an answer that jumps out at me for this one. For the immediate future, don't sweat it if you're not seeing it in GWT. This is assigned to our help desk, and we'll have someone from there investigate more and get back to you, though it might be a few days because of the Thanksgiving holiday (if you don't get an answer today, it may be Monday before we have a chance to respond).
-
If they're not appearing on WMT than you should ignore unless it's an exact duplicated content, then delete
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz cant crawl site?
We're getting an error saying Moz is getting an errors crawling our client's site, but when I've put this though Google Search Console I'm not seeing any issues - any suggestions?
Product Support | | Ramarketingrob0 -
Keyword Ranking Report shows 3 duplicates for each keyword
I have a question about tracked keyword reports. When I extract my data for November for one of my campaigns, there seems to be 3 duplicates of each keyword in the report, each showing different ranking and rank change data. Can you confirm why this happens and how I can tell which the most recent data is? Thanks
Product Support | | John-Clark0 -
"Our crawler was not able to access the robots.txt file on your site."
Hi Mozzers! I've received an error message saying the site can't be crawled because Moz is unable to access the robots.txt. I've spoken to the webmaster and he can't understand why the robot.txt can't be accessed as this seems to be fine: https://k3syspro.com/robots.txt and Google isn't flagging anything up to us. Does anyone know why this may be? Thanks, Matthew
Product Support | | K3Syspro0 -
Duplicate page content on Moz PRO - http://www and http:// problem
Hi Moz community, My Moz Pro writes 555 duplicate content errors, when i click on the page address to check witch page duplicate, it gives me the same page address without WWW. http://www.domain.com/page Duplicated: http://domain.com/page The first thing i checked is my .Htaccess file.. , I found out i have a www redirection that redirect everyone without the "www" to the full domain with the "www" - and it works perfectly on the browser. The code that redirect: RewriteCond %{HTTP_HOST} !^www.
Product Support | | DigitalST
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L] I'm trying to figure out why Moz Pro gives me these errors, Thanks in advanced 🙂0 -
Rogerbot not crawling our site
Has anyone else had issues with Roger crawling your site in the last few weeks? It shows only 2 pages crawled. I was able to crawl the site using Screaming Frog with no problem and we are not specifically blocking Roger via robots.txt or any other method. Has anyone encountered this issue? Any suggestions?
Product Support | | cckapow0 -
Moz Analytics Reports Scheduling
Hi, We have been using Moz now for a few weeks now and we are trying to run reports based on various time frames and can't seem to do so? For example on the if we needed a report from say 17/04/2014- 30/04/2014 how would we achieve this? Or if we wanted to schedule reports based on a calendar month- how do we achieve this? I tried running reports on the 1/05/2014 and received reports for Apr 25 - May 2, 2014. Also on the DASHBAORD- the timeframe only appears to display 'WEEKLY' and the last week of the month- 23th April- 30- Apr. Or will this change once we have more historical data compiled? Thanks,
Product Support | | Faxem0 -
Moz Ranking report help
Any way to show the ranking changes comparing a old dates ranking report- like 4 months ago vs the latest update?
Product Support | | DavidKonigsberg0 -
Why is Moz report showing duplicate content?
Dear Moz Community Our weekly Moz crawl diagnostic repoart is showing a significant increase in "Duplicate Page Content" errors for article pages that have unique content, unique file names, unique META title/descriptions, and unique H1 tags. Where could the duplication be coming from? Thanks for your help.
Product Support | | BoomDialogue690