Block subdomain directory in robots.txt
-
Instead of block an entire sub-domain (fr.sitegeek.com) with robots.txt, we like to block one directory (fr.sitegeek.com/blog).
'fr.sitegeek.com/blog' and 'wwww.sitegeek.com/blog' contain the same articles in one language only labels are changed for 'fr' version and we suppose that duplicate content cause problem for SEO. We would like to crawl and index 'www.sitegee.com/blog' articles not 'fr.sitegeek.com/blog'.so, suggest us how to block single sub-domain directory (fr.sitegeek.com/blog) with robot.txt?
This is only for blog directory of 'fr' version even all other directories or pages would be crawled and indexed for 'fr' version.
Thanks,
Rajiv -
Hi Rajiv,
If you post the same content on both FR & EN version:
-
if both are written in English (or mainly written in English) - best option would be to have a canonical pointing to the EN version
Example: https://fr.sitegeek.com/category/shared-hosting - most of the content is in English - so in this case I would point a canonical to the EN version -
if the FR version is in French - you can use the HREF lang tag - you can use this tool to generate them, check here for common mistakes and doublecheck the final result here.
Just some remarks:
-
partially translated pages offer little value for users - so it's best to fully translate them or only refer to the EN version
-
I have a strong impression that the EN version was machine translated to the FR version. (ex. French sites never use 'Maison' to link to the Homepage - they use Acceuil). Be aware that Google is perfectly capable to detect auto-translated pages and they consider it to be bad practice (check this video of Matt Cutts - starts at 1:50). So you might want to invest in proper translation or proofreading by a native French speaker.
rgds
Dirk
-
-
Thanks Dirk,
we will fix the issue as you suggested.
Could you explain more on duplicate content if we post articles on both 'FR' and 'EN' versions?
Thanks,
Rajiv
-
Just to add to this, if your subdomain has more than /blog on it, and you only want to block /blog, change Dirk's robots.txt to:
User-agent: Googlebot
Disallow: /blogor to block more than just google:
User-agent:*
Disallow: /blog -
The easiest way would be to put the robots.txt in the root of your subdomain & block access for search engines
User-agent: Googlebot
Disallow: /If you subdomain & the main domain are sharing the same root - this option is not possible. In that case, rather than working with robots.txt I would add a canonical on each page pointing to the main domain, or block all pages in the header (if this is technically possible)
You could also check these similar questions: http://moz.com/community/q/block-an-entire-subdomain-with-robots-txt and http://moz.com/community/q/blocking-subdomain-from-google-crawl-and-index - but the answers given are the same as the options above.
Apart from the technical question, qiven the fact that only the labels are translated, these pages make little sense for human users. It would probably make more sense to link to the normal (English) version of the blog (and put (en Anglais) next to the link.
rgds,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I exclude my knowledge center subdomain from indexing?
We have a very large Knowledge center that is indexed. Is there any reason I should not exclude this subdomain from indexing? Thank you
Intermediate & Advanced SEO | | NikCall2 -
Can my affiliate subdomain hurt in any way?
Hello everyone, My main website is: http://www.virtualsheetmusic.com Whereas the above site's related "affiliate" website is located on the subdomain below: http://affiliates.virtualsheetmusic.com I was wondering if having that "affiliate section" on a subdomain could affect the main website negatively in some way... or would be better to put it in a sub-folder on the main website, or even on a totally different domain. Thanks in advance for any advice!
Intermediate & Advanced SEO | | fablau0 -
URLs with parameters + canonicals + meta robots
Hi Moz community! I'm posting a new question here as I couldn't find specific answer to the case I'm facing. Along with canonical tags, we are implementing meta robots on our pages (e-commerce website with thousands of pages). Most of the cases have been covered but I still have one unanswered case: our products are linked from list pages (mostly categories) but they almost always include a tracking parameter (ie /my-product.html?ref=xxx) products urls are secured with a canonical tag (referring only to the clean url /my-product.html) but what would be the best solution regarding the meta robots? For now we opted for a meta robot 'noindex, follow' for non canonical urls (so the ones unfortunately linked from our category/list pages), but I'm afraid that it could hurt our SEO (apparently no juice is given from URLs with a noindex robots), and even maybe prevent bots from crawling our website properly ... Would it be best to have no meta robots at all on these product urls with parameters? (we obviously can't have 'index, follow' when the canonical ref points to another url!). Thanks for your help!
Intermediate & Advanced SEO | | JessicaZylberberg0 -
Which one is better, a brand new subdomain or a second-level directory with PR 4
Hey, all SEOers! May I ask you a question about subdomain and second-level directory? Our website is about software, so we write many posts about how to use this software solve problems, and then use these posts to get ranks (we don't use the page of software to get ranks). And all the posts we wrote are listed under the second-lever directory, just like: www.xxx.com/support/ . But at this moment our boss want to list all the posts to the subdomain like support.xxx.com. By the way, the second-level directory is a page with PR 4, and the subdomain is brand new, even it doesn't exist now. So here is my question: should we list all the posts to support.xxx.com? If we choose to do like this, this will effect the speed of Google index, and we will take more time to build links for XXX.com and support.XXX.com? Any answer will be appreciated and thank you advance! to get rank instead of ranking the page of product,
Intermediate & Advanced SEO | | Vicky28850 -
Best Way to Optimize 38 Local Directory Listing In Major Directories
Hi Folks, I am trying to figure out the best way to get our company's 38 U.S. locations in the major local directories. To start, I'd like to get us listed in the major ones: Google, Yahoo, Bing, and Yelp. I do have the resources myself here on staff to do everything manually. So, I don't necessarily need a service like Yext (but would also like any opinions on that offering if anyone can offer it). But, from what I know in the past, every time you try to claim a local listing within each platform, you have to confirm your existence there somehow - whether it be by a mailed postcard or some sort of automated call they give you. Considering that we want to manage all social and local platforms here at corporate, how can we do this? I am not physically at these locations, but I'm sure this is possible to manage everything through one account. The addresses will be local, but the phone numbers on each local profile will route to our customer service here at corporate because the local locations are mostly administrative. In other words, businesses is booked through corporate and carried out at local destinations. Thoughts/Comments?
Intermediate & Advanced SEO | | CSawatzky
I want to do what's best for SEO and also dont' want to harm anything or our link equity. Thanks,
Pedram0 -
Odd Results Moving Subdomain Content onto Main Domain
Hi forum! On Thursday night (12/6/12) we moved a page (and all the linking product pages) from our subdomain, mailing-list.consumerbase.com, to our main domain, www.consumerbase.com/mailing-lists.html Shockingly, today I search for "mailing lists" (our #1 target keyword) and we're on the first page! This page never has not ranked well for this keyword in the past. The problem is, the link displaying on Google is our old mailing-list.consumerbase.com subdomain URL. Did moving this content from the new subdomain to our old, well-established domain cause it to appear better in search? Or, since the URL is on the subdomain, did Google just finally get around to indexing that page? Thanks!
Intermediate & Advanced SEO | | Travis-W0 -
Should we stream videos from a subdomain or subfolder?
This is causing some hand-wrangling and I need some help. Let's say we were developing rich video content for our products and our agency is hosting the content on a new external server. There are already plans to link to these videos from product detail pages, category pages, and landing pages on our main website: www.example.com. Would it be better to create a new subdomain or to use a subfolder with a reverse proxy technique for this video content? It's not truly a microsite and will not have a gallery page to navigate the various videos. For example, would it be better to use this: video.example.com/ ...or this (executed with a reverse proxy😞 www.example.com/video/ Of course, regardless of whether this new content will live on a subdomain or within a subfolder, we plan on creating a video Sitemap using guidance from Justin Hammack in this terrific post.
Intermediate & Advanced SEO | | Ryan-Ricketts1 -
Subdomains for niche related keywords
I wanted to know how efficient using a subdomain is, taking in consideration all the updates Google has made lately. I am looking to use a subdomain for a well branded website for a niche specific part of their website. The subdomain will end-up having more than 100 pages. I'd like to see in what cases do you guys recommend using a subdomain? How to get the same benefit out of a subdomain as i am getting from the actual main domain?
Intermediate & Advanced SEO | | CMTM0