Can I robots.txt an entire site to get rid of Duplicate content?
-
I am in the process of implementing Zendesk and will have two separate Zendesk sites with the same content to serve two separate user groups (for the same product-- B2B and B2C). Zendesk does not allow me the option to changed canonicals (nor meta tags). If I robots.txt one of the Zendesk sites, will that cover me for duplicate content with Google? Is that a good option? Is there a better option.
I will also have to change some of the canonicals on my site (mysite.com) to use the zendesk canonicals (zendesk.mysite.com) to avoid duplicate content. Will I lose ranking by changing the established page canonicals on my site go to the new subdomain (only option offered through Zendesk)?
Thank you.
-
Just disallow in Robots. No need to do anything else.
-Andy
-
What if the site is not live yet?
-
Hi,
I do mean use robots.txt to block crawlers.
What you need to do is first noindex the site in question and then after a period of time, you can disallow it via the robots.txt.
The reason you do it this way is because right now you will have pages from this site indexed in Google - these need to be removed first. You can either do this with the noindex META and wait for Google to spider the site and action all of the noindex requests, or to speed things up, noindex the page and then remove it with Webmaster Tools.
If you don't do this, you are then just blocking the site from Google ever seeing it, so you will probably find that pages remain in the index - which you don't want as this is duplicate content.
-Andy
-
Thank you. I do mean use robots.txt to block crawlers.
-
Hi there!
Just for clarification, I'm really not sure what you mean by "robots.txt-ing" the site. Do you mean, should you use robots.txt to block crawlers from accessing the entire site? That would be fine, if you're not concerned about that site never ranking, ever.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate content? - Ecommerce reviews loading the same products on every page
Hello there! I use a plugin on my ecom site that shows customer reviews - not product reviews but general shopping experience reviews. The plugin also loads links and short descriptions of products those customers bought. Having installed it site-wide, on every page there are short descriptions of the same products. Of course, as people leave new reviews the content changes (but it doesn't happen very often). So the question is: Is having links and short descriptions of the same products on every page harmful for SEO in this case? I'd be grateful for any insight into this matter.
On-Page Optimization | | thpchlk0 -
I have an eCommerce Site with in some cases, 100s of versions of the same product. How do I avoid "duplicate content" without writing literally 100s of unique product descriptions for the exact same product?
For instance, one item where the only difference is the Sports Team Logo is different, etc... or It comes in a variety of color Variants. I'm using Shopify.
On-Page Optimization | | pstone291 -
Duplicate content shown in Google webmaster tools for 301 redirected URLs.
Why does Google webmaster tools shows 5 URLs that have been 301 redirected as having duplicate meta descriptions?
On-Page Optimization | | Madlena0 -
Duplicate Content
Part of a site I am working on, features many different bags in all thicknesses colors and sizes. I'm getting an error when some pages have different content like different thicknesses. The only differences between the pages are a single digit - but in trash bags that makes it a whole different product! I can't do a canonical because it's not the same. For example: http://www.plasticplace.net/index.php?file=productdetail&iprod_id=274 and http://www.plasticplace.net/index.php?file=productdetail&iprod_id=268 Any ideas?
On-Page Optimization | | EcomLkwd0 -
Where Can I Find Freelance Writer? Content free from Copyscape
Hi, Anyone know where i can get freelance worker that writes
On-Page Optimization | | chanel27
about 500 to 600 words original article that is free from Copyscape? And also how much does it cost in general?0 -
Best practice to solve this Unique duplicate page content issue?
I just got Seomoz Pro (it's awesome!), and when I did a campaign for my website I discovered that I have a big issue with duplicate page content (as well as titles). The Crawl Diagnostics Summary told me I have 196 Crawl Errors Found (I had a total of 362 pages crawled on my site), and as much as 160 of these was duplicate page content. Which to me sounds like a big problem, correct me if I'm wrong (I'm very new to SEO). So our website is an ecommerce that sells greeting cards. The unique part about our platform is that we offer the customer to make a customization of the cards.
On-Page Optimization | | danielpett
Let me walk you through each step a customer takes so you fully understand: They find a card they like and visit the product page of that card (just like on any ecommerce store.) They then decide they want to buy it. There is no "Add to cart" button, they will instead click on a "customize the card" button. 3) This takes them to a step by step process of customizing the card. They change the name on the front of the greeting card so it says for example: "Happy Birthday Katy!". And then adds a personal text on the inside of the card. They then add an delivery address and when it should be delivered. After that they proceed to checkout and it's all done. This is my website (it's in Swedish): loveday.se - it will take you to a product page so that you can click the green button and see what I mean with the customization pages. Hopefully it helps even though it's in Swedish. My issue starts at the customization part of the site (the bolded step above), as I can see the permalinks in the diagnostics I got.
This step-by-step process looks exactly the same with every card in the store. Same call-to-action headline, same descriptive text etc. The only difference is a JPEG-file with the unique greeting card design. So, what is your take on this? Let me know if I was unclear about something. Any help or advice is greatly appreciated.0 -
Hi.. Can a E-commerce site have a Google Authorship.
Hi Can a E-commerce site have a Google Authorship, and if yes i have learned google requires a face for Google Authorship, as they are applying Facial recognition with the authorship. If So, how can an e-commerce website have an individual's face. ?
On-Page Optimization | | usef4u0 -
Duplicate content http:// something .com and http:// something .com/
Hi, I've just got a crawl report for a new wordpress blog with suffusion theme and yoast wordpress seo module and there is duplicate content for: http:// something .com and http:// something .com/ I just can't figure out how to handle this. Can I add a redirect for .com/ to .com in htaccess? Any help is appreciated! By the way, the tag value for rel canonical is **http:// something .com/ **for both.
On-Page Optimization | | DanielSndstrm0