Solve duplicate content issues by using robots.txt
-
Hi,
I have a primary website and beside that I also have some secondary websites with have same contents with primary website. This lead to duplicate content errors. Because of having many URL duplicate contents, so I want to use the robots.txt file to prevent google index the secondary websites to fix the duplicate content issue. Is it ok?
Thank for any help!
-
yes, robots.txt is the bad way, I will try to use canonical tag. Thanks for your help!
-
Using robots.txt is perhaps not the best way of doing it. Using the canonical or a noindex meta tag would likely be best. I think the reasons for this are best summed up in this article which explains, probably better than I could, why robots.txt is not the best way of dealing with duplicate content. Hope this helps.
-
I have tried to use cross-domain canonical, but this is too difficult for me. So, I want to confirm if I use the robots.txt file is ok or not?
Thanks
-
Why not use a cross-domain canonical whereby you reference the pages on your primary website as the canonical version on your secondary websites, thereby eliminating the duplication.
For example on each page that is duplicate on your secondary website you would add the following to the head to reference the primary pages:
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ratings pages are Duplicate Content
This brought up another question. should the review page (which now has a canonical to the item page) be Index,follow? My item review pages are showing up with Duplicate Content errors in MOZ. Here are two examples http://www.americanmusical.com/ItemReview--i-HAM-SK1-LIST http://www.americanmusical.com/ItemReview--i-MAC-203680902-LIST is the problem that the pages contain the same code and questions with very little customer created info?
On-Page Optimization | | dianeb1520 -
Is duplicate content harmful? Example from on my site
I'm not talking about content copied from another site but content unique to a site being used on several pages. I have a delivery tab that has precisely the same content as another product page. This content is on four product pages and the dedicated delivery page. Thanks
On-Page Optimization | | Brocberry0 -
Duplicate Content - Meta Data for International Site Roll Out
Hi All, We have a site targeting Ireland, so all on-page SEO is completed and launched on the Irish site. We are now rolling out this site to the UK...how much of this content & SEO meta data has to be changed for Google to not recognise it as duplicate content? Site structure is as follows: http://www.domain.com/ie-en/ - Irish site http://www.domain.com/uk-en/ - UK site Or will it even be considered duplicate content as we have the uk and Irish signals in the subfolders, will be using geo targeting on webmasters, and will have UK specific addresses and phone numbers? We will be rolling this site out to may more countries so would be great to get this straight from the start so we don't waste time creating many versions of the meta data unnecessarily! Many thanks Emma
On-Page Optimization | | john_Digino0 -
Checking for content duplication against content on your own site.
We are currently trying to rewrite our product descriptions and I'm afraid some of the salespeople that are writing the descriptions are plagiarizing one-another's writing. Is there a content duplication checker that will allow you to check a piece of writing against a specific site rather than all of the web?
On-Page Optimization | | MichealGooden0 -
Duplicate content and the Moz bot
Hi Does our little friend at SEOmoz follow the same rules as the search engine bots when he crawls my site? He has sent thousands of errors back to me with duplicate content issues, but I thought I had removed these with nofollow etc. Can you advise please.
On-Page Optimization | | JamieHibbert0 -
How woud you deal with Blog TAGS & CATEGORY listings that are marked a 'duplicate content' in SEOmoz campaign reports?
We're seeing "Duplicate Content" warnings / errors in some of our clients' sites for blog / event calendar tags and category listings. For example the link to http://www.aavawhistlerhotel.com/news/?category=1098 provides all event listings tagged to the category "Whistler Events". The Meta Title and Meta Description for the "Whistler Events" category is the same as another other category listing. We use Umbraco, a .NET CMS, and we're working on adding some custom programming within Umbraco to develop a unique Meta Title and Meta Description for each page using the tag and/or category and post date in each Meta field to make it more "unique". But my question is .... in the REAL WORLD will taking the time to create this programming really positively impact our overall site performance? I understand that while Google, BING, etc are constantly tweaking their algorithms as of now having duplicate content primarily means that this content won't get indexed and there won't be any really 'fatal' penalties for having this content on our site. If we don't find a way to generate unique Meta Titles and Meta Descriptions we could 'no-follow' these links (for tag and category pages) or just not use these within our blogs. I am confused about this. Any insight others have about this and recommendations on what action you would take is greatly appreciated.
On-Page Optimization | | RoyMcClean0 -
Duplicate Page Content Issue
For one of our campaigns, we have 164 errors for Duplicate Page Content. We have a website where much of the same content lives in two different places on their website. The information needs to be accessible from both areas. What is the best way to tackle this problem? Is there anything that can be done so these pages are not competing against one another? If the only solution is to edit the content on one of the pages, how much of the content has to be different? Is there a certain percentage to go by? Here is an example of what I am referring to: 1.) http://www.valleyorthopedicassociates.com/services/foot-center/preventing-sprains-and-strains 2.) http://www.valleyorthopedicassociates.com/patient-resources/service/foot-and-ankle-center/preventing-sprains-and-strains
On-Page Optimization | | cmaseattle1 -
Duplicate page title and content
Hello, I have an ecommerce store where we offer many similar products, and the main difference could be the color or memory storage. Due to this reason my main problem appears to be be duplicate page title and content. What is the best way to correct this issue? I cant make them different neither. I always include this particular difference in title or description. I guess it is not enough? any way to fix it? thanks!
On-Page Optimization | | tolyadem10