SEOMoz Crawler vs Googlebot Question
-
I read somewhere that SEOMoz’s crawler marks a page in its Crawl Diagnostics as duplicate content if it doesn’t have more than 5% unique content.(I can’t find that statistic anywhere on SEOMoz to confirm though).
We are an eCommerce site, so many of our pages share the same sidebar, header, and footer links. The pages flagged by SEOMoz as duplicates have these same links, but they have unique URLs and category names. Because they’re not actual duplicates of each other, canonical tags aren’t the answer. Also because inventory might automatically come back in stock, we can’t use 301 redirects on these “duplicate” pages.
It seems like it’s the sidebar, header, and footer links that are what’s causing these pages to be flagged as duplicates. Does the SEOMoz crawler mimic the way Googlebot works? Also, is Googlebot smart enough not to count the sidebar and header/footer links when looking for duplicate content?
-
Google bot definitely won't ding you for duplicate content based upon you sidebar, header, and footer links. I've managed several ecommerce sites and have never had this problem.
However, it may still ding you for duplicate content if you have several matching pages with you main column being the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTTPS vs HTTP
A few months ago we switched all product page urls in our e-commerce site to https: Recently working on the site I was appalled at how slow our pages were loading and on investigating further with our hosting partner they advised to switch back to http instead of https for all of the site content to help page speed. We obviously still need to use https in the cart and check-out. I think that that Google will be pushing all commerce web pages to https but for now I need to improve page load speed. Will the switch back from https to http impair our keywords? https://www.silverandpewtergifts.com/
Technical SEO | | silverpewter0 -
Block bad crawlers
Hi! how are you? I've been working on some of my sites, and noticed that i'm getting lots of crawls by search engines that i'm not intereted in ranking well. My question is the following: do you have a list of 'bad behaved' search engines that take lots of bandwidth and don´t send much/good traffic? If so, do you know how to block them using robots.txt? Thanks for the help! Best wishes, Ariel
Technical SEO | | arielbortz0 -
Schema Address Question
I have a local business with a contact page that I want to add schema markup to. However, I was wondering if having the address with schema info on the contact page instead of the home page has any adverse effects on the rich snippet showing up in search. There's no logical place to add schema for a local business on the home page, so having it on the contact page—not in the footer or sidebar—is the only option.
Technical SEO | | DLaw0 -
Drupal Question
So on our site we have a plugin for our fan gallery. The issue is that I am getting a lot of duplication errors and it's saying the URL is too long and all the errors are coming from the Fan Gallery, which has over 8,000 errors. It seems to be pulling a long form query URL that has over 100 characters. You can't physically see it on the site, but the crawlers can. Anyway I'm trying to figure out a fix for this. One method would be to just stop those pages from being crawled, but I would hate to do that as the fan gallery for us would be a great source of links and content. So I'm wondering if anyone else has had an issue with these types of plugins before where the user can upload a photo or do a video embed and then it submits to the site. If you have a better method please let me know. I usually work on E-comm platforms so my experience with drupal is limited.
Technical SEO | | KateGMaker0 -
Domain tld question
Hi all, I have a question regarding the ranking of exact match tld which is co.uk Currently I have a .com domain with PR of 3 and the problem is that it have one word in front of my desired keyword, so it's not exact match. I have managed to buy an exact match but it's co.uk The questions are: Will a co.uk rank better for UK than .com domain I am reading at SEOMOZ that exact match domain value is getting lower, so is it worth to redirect my current .com domain to co.uk just to get rid of that one word and start all over again with exact match. Thanks
Technical SEO | | VasilTasev0 -
Sub pages Vs subdomain Pagerank flow.
I Have Question about Pagerank flow:
Technical SEO | | tommytai
If ihave a site :domain.com and i have 2 solutions like:
Solution #1: Quote: | domain.com/blog and domain.com/video if i try to do :
Root domain only link to
domain.com/video and
domain.com/blog | Solution #2: Quote: | Root domain only link to
video.domain.com and
blog.domain.com | So <acronym title="Google PageRank">Pr</acronym> domain.com/blog = <acronym title="Google PageRank">Pr</acronym> blog.domain.com ?
and <acronym title="Google PageRank">PR</acronym> domain.com/video = <acronym title="Google PageRank">Pr</acronym> video.domain.com ? And don't know why a subdomain of blogspot or Wordpress ranking easier than a new domain like:keyword.wordpress.com and keyword.com So What Wordpress pass to keyword.wordpress.com ?0 -
Backlink: External blog Vs. Internal blog. Which is the best?
Hi, some weeks ago a created a blog: mykeyword.wordpress.com Some one told me that it has got more trust that a "normal" www.mykeyword.com
Technical SEO | | Greenman
Is it true? So, i wrote some articles and dropped a guide (linking inside to mysite.com) to blog. My question is:
Right now i'm writing a lot of article ad i'm looking for the best channel where publish my content (post with link inside). My focus is improving quantity and quality of backlinks. Which way must i use? 1. Use my mykeyword.wordpress.com (give freshness to blog and new backlink)
2. Create ad internal blog mysite.com/blog and add article (without link?)
3. "Don't lose time" - Put new article only in external blog that will link to my site. I must manage a lot of new sites and i should increase SERP position. So, i have to choose the right way right now. Thanks 😉0 -
302 vs. a href="nofollow"
we came across one thing the we did not asked to programm by our intention. we have a magento shop and on the produktpage we have those "compare" buttons. these link have a session id and the follow a 302 back onto the same page. so i beleive the idea is that google will just not follow 302s and thats it. so my questions is: is this right what we beleive if so why is a 302 better compared to a a href="nofollow" ???
Technical SEO | | kynop0