Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What is Considered Duplicate Content by Crawlers?
-
I am asking this because I have a couple of site audit tools that I use to crawl a site I work on every week and they are showing duplicate content issues (which I know there is a lot on this site) but some of what is flagged as duplicate content makes no sense.
For example, the following URL's were grouped together as duplicate content:
|
https://www.firefold.com/contact-us
|
| https://www.firefold.com/sale |
|
|
How are these pages duplicate content? I am confused on what site audit tools are considering duplicate content.
Just FYI, this is data from Moz crawl diagnostics but SEMrush site auditor is giving me the same type of data.
Any help would be greatly appreciated.
Ryan
-
Yea I just started working on this site. I haven't used Moz Analytics much so just wanting to see how their crawler crawls pages.
And yes I agree, there are a lot of BIG BIG BIG issues with this site.
I got a large workload over the next few months haha.
-
I would add that there's is no text on any of those three pages - any "text" one would see there is actually just embedded in an image - which is a huge issue for a number of reasons:
- Search engines see that there's no text - a big no-no.
- You're getting practically no SEO value from the content that would be there, even if there isn't much.
- It's heavier this way - which makes load times slower.
I want to clarify that there are many, bigger issues with these pages - but as your question concerns only duplicate content, I'll leave all of that out for the time being. To summarize, Google, Yahoo, and Bing are just seeing some duplicate banners, sidebars, etc. and then some images in the body of your pages. Hence, duplicate content.
-
Thanks for that information.
It makes sense looking at the data and pages from that perspective.
-
Hi Ryan!
Our crawler will flag pages that have at least 90% similarity in the entire source code of the site so not just the body.
The way you want to interpret the report is the contact-us page has 35 duplicates, so "gabe" and "sale" are not dupes of each other in this section but are only each a duplicate of "contact-us". Those URLs might appear with their own duplicates of the same pages further down in the report.
While on the front end the pages do not appear to be similar. The issue is likely with the amount of javascript code on those pages.
Our crawler cannot read javascript so we are likely only able to see the template of the page. Other search tools are probably seeing the same thing as it returns 79% similarity using this tool: http://www.freebulkseotools.com/similar-page-checker-tool.php
I can't provide much insight from a dev perspective but hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved New Moz Bar not working in my domain
Hello, I have installed the new version of Moz Bar in Google Chrome.
Moz Bar | Nov 16, 2024, 8:06 AM | Alberto D.
My problem is that it analyzes all domains except mine: https://elchesemueve.com
I also can't find my country (Spain) to configure the site.0 -
Unsolved How Should I Increase DA of my Website?
Google itself has been troubled by ranking factors. On top of that, the https://moz.com/ has worn a new garland of domain authority. It is even more regrettable when a website DA suddenly decreased. As recently on 22nd September, my website's DA was 16. As usual, on September 23, I posted a new informational blog related to QuickBooks antivirus exclusions on my website. The very next day my website DA was decreased. Can anyone who knows SEO help me understand why this always happens specially to me every time? I read the whole forum and all I found out is how my site's DA is decreased.
Moz Bar | Oct 26, 2024, 4:48 PM | ravigaurav
Can anyone let us know whether we can increase the Domain Authority of our website only by creating backlinks or there can be some other activities also?2 -
Unsolved New Google SERP Now Lazy Loading
Hello, Google has changed the way it displays it's SERPs with lazy loading new listings. Because of this the Moz Chrome Extension no longer shows results past first results. I also often find that when a site is ranked above the Maps, it doesn't display results. Is there a fix for this?
Moz Bar | Mar 5, 2024, 5:22 PM | HercMagnus0 -
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | May 8, 2024, 7:32 PM | jbcorcoran6 -
Spam score of 28 %
Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,
Moz Bar | Apr 8, 2024, 4:50 PM | seoanalytics1 -
Www and non www / duplicate content / redirects / www resolve issue
I am not getting docked for these specific errors, but I am getting docked for 1 page has a WWW resolve issue and 1 wrong URL in the sitemap... (SEM Rush) but when I use moz, it's not showing any issues. So I have these things set up so far: In .htaccess i have a command that removes the www. 301 redirect from www version to the non www (homepage) canonical on index.html pointing to non www version, I also set up a canonical tag for each page on the site search console with non www, www, https www, https non www all set to non www preference. Also, when I fetch the www version in google search console it says it's being 301 redirected to non www version which is basically what I want.Is there anything that i'm missing? These errors on SEM Rush are giving me anxiety lol.
Moz Bar | Nov 19, 2016, 5:35 PM | donnieath1 -
Duplicate Content on Website with Multiple Locations
Hi there, I've spent hours reading posts on duplicate content and googling this but I'm still not sure what to do. We created a site that has two WP installs for a company with two different locations - the landing page is website.com and links to WP install 1 (website.com/city1), and WP install 2 (website.com/city2). They specifically wanted two different sites so they could be managed by staff at either location. However some of the pages have the same content - ie. services, policies, etc. so all of those are showing errors for duplicate content. All pages have different city-specific URL's and meta-descriptions but that clearly doesn't help. We can't redirect the "duplicate" pages because then it would take the user to the other city's specific site. Is there anything we can do?? Is this going to significantly damage rankings? Thanks kindly for any help you can provide.
Moz Bar | Dec 15, 2015, 4:29 PM | charlie0071 -
Duplicate page issue
Hi Guys We were recently having trouble with an excessive amount of duplicate page titles. So I asked our web company, at a reasonable expense, to fix the issue which they did. How ever I since note that the issue has returned (please see the attached graph. Could anyone explain to me why this might have happened? I t would be great to have some insight before i go back to them. Thanks again for your help Regards Pete Capture.jpg
Moz Bar | Oct 13, 2013, 11:27 PM | Hardley1110