undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Bar
  4. What is Considered Duplicate Content by Crawlers?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

What is Considered Duplicate Content by Crawlers?

Moz Bar
3
5
934
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • RyanRhodes
    RyanRhodes last edited by Aug 19, 2015, 5:30 PM

    I am asking this because I have a couple of site audit tools that I use to crawl a site I work on every week and they are showing duplicate content issues (which I know there is a lot on this site) but some of what is flagged as duplicate content makes no sense.

    For example, the following URL's were grouped together as duplicate content:

    |

    https://www.firefold.com/contact-us

    |

    https://www.firefold.com/gabe

    | https://www.firefold.com/sale |

    |

    |

    How are these pages duplicate content? I am confused on what site audit tools are considering duplicate content.

    Just FYI, this is data from Moz crawl diagnostics but SEMrush site auditor is giving me the same type of data.

    Any help would be greatly appreciated.

    Ryan

    1 Reply Last reply Reply Quote 0
    • RyanRhodes
      RyanRhodes last edited by Aug 19, 2015, 7:03 PM Aug 19, 2015, 7:03 PM

      Yea I just started working on this site. I haven't used Moz Analytics much so just wanting to see how their crawler crawls pages.

      And yes I agree, there are a lot of BIG BIG BIG issues with this site.

      I got a large workload over the next few months haha.

      1 Reply Last reply Reply Quote 0
      • Lumina
        Lumina @DavidLee last edited by Aug 19, 2015, 6:57 PM Aug 19, 2015, 6:57 PM

        I would add that there's is no text on any of those three pages - any "text" one would see there is actually just embedded in an image - which is a huge issue for a number of reasons:

        1. Search engines see that there's no text - a big no-no.
        2. You're getting practically no SEO value from the content that would be there, even if there isn't much.
        3. It's heavier this way - which makes load times slower.

        I want to clarify that there are many, bigger issues with these pages - but as your question concerns only duplicate content, I'll leave all of that out for the time being. To summarize, Google, Yahoo, and Bing are just seeing some duplicate banners, sidebars, etc. and then some images in the body of your pages. Hence, duplicate content.

        1 Reply Last reply Reply Quote 2
        • RyanRhodes
          RyanRhodes @DavidLee last edited by Aug 19, 2015, 6:56 PM Aug 19, 2015, 6:56 PM

          Thanks for that information.

          It makes sense looking at the data and pages from that perspective.

          1 Reply Last reply Reply Quote 0
          • DavidLee
            DavidLee last edited by Sep 2, 2015, 9:06 AM Aug 19, 2015, 6:14 PM

            Hi Ryan!

            Our crawler will flag pages that have at least 90% similarity in the entire source code of the site so not just the body.

            The way you want to interpret the report is the contact-us page has 35 duplicates, so "gabe" and "sale" are not dupes of each other in this section but are only each a duplicate of "contact-us". Those URLs might appear with their own duplicates of the same pages further down in the report.

            While on the front end the pages do not appear to be similar. The issue is likely with the amount of javascript code on those pages.

            Our crawler cannot read javascript so we are likely only able to see the template of the page. Other search tools are probably seeing the same thing as it returns 79% similarity  using this tool: http://www.freebulkseotools.com/similar-page-checker-tool.php

            I can't provide much insight from a dev perspective but hope this helps!

            RyanRhodes Lumina 2 Replies Last reply Aug 19, 2015, 6:57 PM Reply Quote 2
            • 1 / 1
            1 out of 5
            • First post
              1/5
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • Josh_Eligon

              Unsolved MozBar 2024 How it Works Now

              mozbar 2024

              I am a premium user and i am still not able to use mozbar
              ec22b5ba-4430-4fc7-9825-7fce7b1ca300-image.png I don't know how its works.

              Moz Bar | Nov 21, 2024, 3:53 AM | Josh_Eligon
              0
            • HercMagnus

              Unsolved New Google SERP Now Lazy Loading

              serp moz local

              Hello, Google has changed the way it displays it's SERPs with lazy loading new listings. Because of this the Moz Chrome Extension no longer shows results past first results. I also often find that when a site is ranked above the Maps, it doesn't display results. Is there a fix for this?

              Moz Bar | Mar 5, 2024, 5:22 PM | HercMagnus
              0
            • rajas2019

              How Much Time It Will Take To Lower the Spam Score?

              I'm facing an issue with my website. Due to little to no knowledge about link building and backlink, i created backlinks without checking the quality and spam score of the sites. Now there are many sites linking to my website but the overall spam score is very high of my website and my domain is reviewheart.com that i'm talking about. I have created a list and disavowed all he poor linking domains but still no improvement shown. Can anyone have the idea how much time moz will take to show the updated spam score as i have disavowed all the low quality spammed linking site?

              Moz Bar | Jul 4, 2024, 9:36 AM | rajas2019
              2
            • seoanalytics

              Spam score of 28 %

              Hello, I checked my website spam score in link explorer and found out it has got 28 %. What do you recommend I do ? should I ask for all the links to be removed (even though some of them are no followed) Is it possible that google has penalised me because of that score and de-ranked me ? Thank you,

              Moz Bar | Apr 8, 2024, 4:50 PM | seoanalytics
              1
            • KanikaG

              Keyword Ranking Report is Different than Real Result on Google

              Hi Guys,
              I am getting MOZ reports on a few keywords and that is good But when My client search for the same keywords he doesn't get the same rank, not even on that page. Let me explain in more details:
              I have a keyword that shows on rank #5 in Google Mobile for a Local location for the current week. The client is from the same location as well. But when client search for that keyword from his mobile he doesn't get the keyword listed on that page, not even on other pages. I checked with other online keyword rank checker tool and all are showing the result similar to the Moz ranking report. It's really frustrating for my client. Google analytics shows that mobile users and organic reach are increasing so I am sure MOZ and other tools are showing us correct result. What could be the possible reason that he is not getting the same result as showing by MOZ? I am not from his location so how can I check the keyword result for the selected city using MOZ that is accurate? Any help would be highly appreciated. Please help. Thanks

              Moz Bar | Apr 29, 2024, 12:13 PM | KanikaG
              1
            • charlie007

              Duplicate Content on Website with Multiple Locations

              Hi there, I've spent hours reading posts on duplicate content and googling this but I'm still not sure what to do. We created a site that has two WP installs for a company with two different locations - the landing page is website.com and links to WP install 1 (website.com/city1),  and WP install 2 (website.com/city2). They specifically wanted two different sites so they could be managed by staff at either location. However some of the pages have the same content - ie. services, policies, etc. so all of those are showing errors for duplicate content. All pages have different city-specific URL's and meta-descriptions but that clearly doesn't help. We can't redirect the "duplicate" pages because then it would take the user to the other city's specific site. Is there anything we can do?? Is this going to significantly damage rankings? Thanks kindly for any help you can provide.

              Moz Bar | Dec 15, 2015, 4:29 PM | charlie007
              1
            • Essential-Pest

              Duplicate Page and Title Issues

              On the last crawl, we received errors for duplicate page titles and some duplicate content pages. Here is the issue: We went through our page titles that were marked as duplicate and changed them to make sure their titles were different. However, we just received a new crawl this week and it is saying there are even more duplicate page title errors detected than before. We're wondering if this is a problem with just us or if it has been happening to other Moz users. As for the duplicate content pages, what is the best way to approach this and see what content is being looked at as a "duplicate" set?

              Moz Bar | Apr 29, 2020, 9:09 PM | Essential-Pest
              0
            • Hardley111

              Duplicate page issue

              Hi Guys We were recently having trouble with an excessive amount of duplicate page titles. So I asked our web company, at a reasonable expense, to fix the issue which they did. How ever I since note that the issue has returned (please see the attached graph. Could anyone explain to me why this might have happened? I t would be great to have some insight before i go back to them. Thanks again for your help Regards Pete Capture.jpg

              Moz Bar | Oct 13, 2013, 11:27 PM | Hardley111
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.