SEOMoz Crawl Diagnostic indicates duplicate page content for home page?
-
My first SEOMoz Crawl Diagnostic report for my website indicates duplicate page content for my home page. It lists the home page URL Page Title and URL twice.
How do I go about diagnosing this?
Is the problem related to the following code that is in my .htaccess file? (The purpose of the code was to redirect any non "www" backlink referrals to the "www" version of the domain.)
RewriteCond %{HTTP_HOST} ^whatever.com [NC]
RewriteRule ^(.*)$ http://www.whatever.com/$1 [L,R=301]Should I get rid of the "http" reference in the second line?
Related to this is a notice in the "Crawl Notices Found" -- "301 Permanent redirect" which shows my home page title as "http://whatever.com" and shows the redirect address as http://http://www.whatever.com/
I'm guessing this problem is again related to the redirect code I'm using.
Also...
The report indicates duplicate content for those links that have different parameters added to the URL i.e. http://www.whatever.com?marker=Blah Blah&markerzoom=13
If I set up a canonical reference for the page, will this fix this?
Thank you.
-
I contacted the help desk as instructed an was told:
"I took a look at the campaign and it looks like our crawler can't parse the 301 redirect you have in place on the main page. The reason for this is the redirect in place, adds two https when rogerbot tries to crawl through it. Roger can’t parse the redirect as is, but it can identify it (as it did in your notice’s report on the crawl diagnostics page). This isn't a problem for browsers since they are made to ignore redirects of this nature. Crawlers on the other hand have a strict code to follow and can't follow redirects like that. When I load up your site [mywebsite.com] right now, it redirects to www.[mywebsite].com. Try creating a new campaign under the domain you are redirecting to, this should clear any issues up."
And so I did that and it looks like that worked after the new crawl, however I then set up another campaign for another website I manage being sure to use the "www" in front of the domain and got the same problem again -- the home page appears twice as duplicate content.
So I'm back to asking my primary question again: What is the definitive redirect code to use to convert a non "www" request to a "www" request? The same redirect code mentioned in my first post is being used on all of my sites.
-
Hi Perry,
In your browser, are you seeing things redirect to the double http? I think there's a bug in the crawl tools that's causing some false errors right now. Before you go work on the redirect file, could you send an email to help@seomoz.org to first make sure we're not the ones that messed up?
Thanks!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Pages , Do they matter ?
I have been told buy the company who created my site that duplicate page warning are not a problem ? my site is small and only has 50 pages ( including product pages etc ) yet the crawl shows over 6500 duplicate pages am I right to be concerned?
Technical SEO | | Gardening4you0 -
Would Google Call These Pages Duplicate Content?
Our Web store, http://www.audiobooksonline.com/index.html, has struggled with duplicate content issues for some time. One aspect of duplicate content is a page like this: http://www.audiobooksonline.com/out-of-publication-audio-books-book-audiobook-audiobooks.html. When an audio book title goes out-of-publication we keep the page at our store and display a http://www.audiobooksonline.com/out-of-publication-audio-books-book-audiobook-audiobooks.html whenever a visitor attempts to visit a specific title that is OOP. There are several thousand OOP pages. Would Google consider these OOP pages duplicate content?
Technical SEO | | lbohen0 -
Magento Multistore and Duplicate Content
Hey all, I am currently optimizing a Magento Multistore running with two store views (one per language). Now when I switch from one language to another the urls shows: mydomain.de/.../examplepage.html?___store=german&___from_store=english The same page can also be reached by just entering mydomain.de/.../examplepage.html The question is: Does Google consider this as Duplicate Content or it it nothing to worry about? Or should I just do a dynamic 301 redirect from the 1st version to the 2nd? I read about some hacks posted in diferent magento forums but as I am working for a customer I want to avoid hacks. Also setting "Add Store Code to Urls" didn't help.
Technical SEO | | dominator0 -
If two websites pull the same content from the same source in a CMS, does it count as duplicate content?
I have a client who wants to publish the same information about a hotel (summary, bullet list of amenities, roughly 200 words + images) to two different websites that they own. One is their main company website where the goal is booking, the other is a special program where that hotel is featured as an option for booking under this special promotion. Both websites are pulling the same content file from a centralized CMS, but they are different domains. My question is two fold: • To a search engine does this count as duplicate content? • If it does, is there a way to configure the publishing of this content to avoid SEO penalties (such as a feed of content to the microsite, etc.) or should the content be written uniquely from one site to the next? Any help you can offer would be greatly appreciated.
Technical SEO | | HeadwatersContent0 -
Thin/Duplicate Content
Hi Guys, So here's the deal, my team and I just acquired a new site using some questionable tactics. Only about 5% of the entire site is actually written by humans the rest of the 40k + (and is increasing by 1-2k auto gen pages a day)pages are all autogen + thin content. I'm trying to convince the powers that be that we cannot continue to do this. Now i'm aware of the issue but my question is what is the best way to deal with this. Should I noindex these pages at the directory level? Should I 301 them to the most relevant section where actual valuable content exists. So far it doesn't seem like Google has caught on to this yet and I want to fix the issue while not raising any more red flags in the process. Thanks!
Technical SEO | | DPASeo0 -
Duplicate Page Issue
Dear All, I am facing stupid duplicate page issue, My whole site is in dynamic script and all the URLs were in dynamic, So i 've asked my programmer make the URLs user friendly using URL Rewrite, but he converted aspx pages to htm. And the whole mess begun. Now we have 3 different URLs for single page. Such as: http://www.site.com/CityTour.aspx?nodeid=4&type=4&id=47&order=0&pagesize=4&pagenum=4&val=Multi-Day+City+Tours http://www.tsite.com/CityTour.aspx?nodeid=4&type=4&id=47&order=0&pagesize=4&pagenum=4&val=multi-day-city-tours http://www.site.com/city-tour/multi-day-city-tours/page4-0.htm I think my programmer messed up the URL Rewrite in ASP.net(Nginx) or even didn't use it. So how do i overcome this problem? Should i add canonical tag in both dynamic URLs with pointing to pag4-0.htm. Will it help? Thanks!
Technical SEO | | DigitalJungle0 -
I have 15,000 pages. How do I have the Google bot crawl all the pages?
I have 15,000 pages. How do I have the Google bot crawl all the pages? My site is 7 years old. But there are only about 3,500 pages being crawled.
Technical SEO | | Ishimoto0 -
Different TLD's same content - duplicate content? - And a problem in foreign googles?
Hi, Operating from the Netherlands with customers troughout Europe we have for some countries the same content. In the netherlands and Belgium Dutch is spoken and in Germany and Switserland German is spoken. For these countries the same content is provided. Does Google see this as duplicate content? Could it be possible that a german customer gets the Swiss website as a search result when googling in the German Google? Thank you for your assistance! kind regards, Dennis Overbeek Dennis@acsi.eu
Technical SEO | | SEO_ACSI0