SEOMoz Crawler vs Googlebot Question
-
I read somewhere that SEOMoz’s crawler marks a page in its Crawl Diagnostics as duplicate content if it doesn’t have more than 5% unique content.(I can’t find that statistic anywhere on SEOMoz to confirm though).
We are an eCommerce site, so many of our pages share the same sidebar, header, and footer links. The pages flagged by SEOMoz as duplicates have these same links, but they have unique URLs and category names. Because they’re not actual duplicates of each other, canonical tags aren’t the answer. Also because inventory might automatically come back in stock, we can’t use 301 redirects on these “duplicate” pages.
It seems like it’s the sidebar, header, and footer links that are what’s causing these pages to be flagged as duplicates. Does the SEOMoz crawler mimic the way Googlebot works? Also, is Googlebot smart enough not to count the sidebar and header/footer links when looking for duplicate content?
-
Google bot definitely won't ding you for duplicate content based upon you sidebar, header, and footer links. I've managed several ecommerce sites and have never had this problem.
However, it may still ding you for duplicate content if you have several matching pages with you main column being the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site not getting indexed by googlebot.
The following question is in regards to http://footeschool.org/. This site is not getting indexed with google(googlebot) This only happens when the user agent is set googlebot. This is a recent issue. We are using DNN as CMS. Are there any suggestion to help resolve this issue?
Technical SEO | | bcmull0 -
Questions about the Sandbox and 301 Redirects
Does the sandbox still exist? What if you have a brand new URL and do a 301 redirect from another website because the name of the service business changed? Thanks for any insight and help.
Technical SEO | | SDSLaw0 -
Too many on-page links vs. UX issue
I am having an issue with many of our pages having too many on-page links. I have gotten many of them below the 100 page limit that is suggested and I understand this is not a critical factor with SEO, but my issue is this: Many important pages I am trying to optimize are buried at a "3rd" level which is actually not accessible from the home page navigation dropdown due to our outdated CMS. I am trying to decide if we should develop our site to display these pages on-hover from the main navigation. This would make a lot of sense since users would find these pages easier, however adding this functionality would increase on-page links by a lot more. So in your opinion, would it be worth it to spend the money to have this functionality developed? Or would it end up hurting our SEO standings?
Technical SEO | | isret_efront0 -
Questionable Referral Traffic
Hey SEOMozers, I'm working with a client that has a suspicious traffic pattern going on. In October, a referral domain called profitclicking.com started passing visits to the site. Almost, in parallel the overall visits decreased anywhere from 35 to 50%. After checking out profitclicking.com more, it promises more traffic "with no SEO knowledge". The client doesn't think that this service was signed up for internally. Regardless, it obviously smells pretty fishy, and I'm searching for a way I can disallow traffic from this site. Could I simply just write a simple disallow statement in the robots.txt and be done with it? Just wanted to see if anyone else had any other ideas before recommending a solution. Thanks!
Technical SEO | | kylehungate0 -
Post vs page in Wordpress?
Hello there, I have a Wordpress site and would like to know if it is better to have 600 posts or 600 pages in terms of efficiency in the site. I would like to publish the content as pages, as I can have subapges,etc... and keep the path: www.website.com/page/subpage1... in terms of good SEO. This structure of using pages rahter than posts allow me to keep the path as stated above (with a category/post path I could not manage in this sense as a pile of articles is displayed although the path category/post in terms of SEO I understand would be good too). Thank you very much for your thoughts here as I would go for a page structure. Antonio
Technical SEO | | aalcocer20030 -
Redirect questions
Hi! A client of mine have created a new site with a new URL structure which they launched the other day. They have done a 301 redirect on all pages on the old site to the start page on the new site. E.g:
Technical SEO | | lojdqvist
www.olddomain.com/subfolder1/index.html -> www.newdomain.com
www.olddomain.com/subfolder2/index.html -> www.newdomain.com I'm thinking of fixing this now so the redirect instead looks someting like this:
www.olddomain.com/subfolder1/index.html -> www.newdomain.com/newsubfolder1/index.html
www.olddomain.com/subfolder1/index.html -> www.newdomain.com/newsubfolder1/index.html Two questions: 1. Is it worth doing the latter kind of redirect in all cases (after all, it involves quite a lot more work compared to the first solution)? or do you recommend the first solution for all redirect projects?
2. Now that they have already done the first solution, is it at all worth amending this to the latter or is everything spoiled now that they have already gone ahead with the first solution? Many thanks in advance!0 -
Robots.txt question
I want to block spiders from specific specific part of website (say abc folder). In robots.txt, i have to write - User-agent: * Disallow: /abc/ Shall i have to insert the last slash. or will this do User-agent: * Disallow: /abc
Technical SEO | | seoug_20050 -
Panda Update Question - Syndicated Content Vs Copied Content
Hi all, I have a question on copied content and syndicated content - Obviously copying content directly form another website is a big no no, but wanted to know how Google views syndicated content and if it views this differently? If you have syndicated content on your website, can you penalised from the lastest Panda update and is there a viable solutiion to address this? Mnay thanks Simon
Technical SEO | | simonsw0