How does robots.txt affect aliased domains?
-
Several of my sites are aliased (hosted in subdirectories off the root domain on a single hosting account, but visible at www.theSubDirectorySite.com) Not ideal, I know, but that's a different issue.
I want to block bots from viewing those files that are accessible in subdirectories on the main hosting account, www.RootDomain.com/SubDirectorySite/, and force the bots to look at www.SubDirectorySite.com instead.
I utilized the canonical meta tag to point bots away from the sub directory site, but I am wondering what will happen if I use robots.txt to block those files from within the root domain.
Will the bots, specifically Google bot, still index the site at its own URL, www.AnotherSite.com even if I've blocked that directory with Disallow: /AnotherSite/ ?
THANK YOU!!!
-
I'm assuming you can't 301-redirect (and that you still need the sub-directory versions to be reachable by humans)? I'm not sure the cross-domain canonical will work completely. I don't have a good example of a sub-folder to root domain canonical implementation. If the "sites" are identical, it should be ok.
Robots.txt is going to depend a bit on how people access those. If there are links to the sub-directory versions, then blocking will cut off that link-juice (and the canonical or a 301 will be better).
Blocking the sub-directory shouldn't automatically block the domain it aliases, too, unless for some reason that sub-directory is the only crawl path Google has to the outside domain. As long as they're crawling the outside domain separately, I think you'll be ok. I'm just not sure if Robots.txt is necessary here.
Sorry, the devil may be in the details on this one. Happy to take a closer look in Private Q&A, if you want to give out some specifics.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recently re-built our site and changed domain. Now I want to go back to old domain - it it a bad idea?
About a year ago I rebuilt our website and changed our domain name. We rent villas in Tuscany, we used to be 'invitationtotuscany.com'. Then I started doing the same in Provence, and in the italian lakes, so i had further sites called invitationtoprovence.com and invitationtotheitalianlakes.com. But maintaining them was awkward and I wanted to have one site. So I put them all onto invitationto.com and did 301s from the old domains and sites. Now I'd dropped off organic search results and I've also realised that invitationto.com is far less clear as a business address. My inclination is to go back to invitationtotuscany.com - Tuscany is still 80% of our business and have the other areas in there too - optimised for SEO for Provence etc. I'm being told its a really bad idea to change domain, 301 the old one, and then revert to the original domain. But I'm suffering anyway, so I wonder if I sjhouldn't just bite the bullet. A lot of my old good backlinks still point to invitationtotuscany.com (BBC, Sunday Times, etc) and the DA is 33 against 22 on the new one.. All help gratefully received! : )
Technical SEO | | DanWrightson0 -
URL Structure On Site - Currently it's domain/product-name NOT domain/category/product name is this bad?
I have a eCommerce site and the site structure is domain/product-name rather than domain/product-category/product-name Do you think this will have a negative impact SEO Wise? I have seen that some of my individual product pages do get better rankings than my categories.
Technical SEO | | the-gate-films0 -
How to use robots.txt to block areas on page?
Hi, Across the categories/product pages on out site there are archives/shipping info section and the texts are always the same. Would this be treated as duplicated content and harmful for seo? How can I alter robots.txt to tell google not to crawl those particular text Thanks for any advice!
Technical SEO | | LauraHT0 -
Purchasing a domain to redirect to a new domain (note same industry) - Black hat or White hat technique?
Hi Everyone, Ok so here is my question. I have a client who sells gourmet tea and gourmet spices. She has a culinary blog. There is a culinary blog that just posted that the website will be shut down in the near future. It has 100% white hat links. Would it be considered black hat to buy the domain and redirect it to my clients blog which is also a culinary blog? I would really like to ask Matt Cutts this question. Does anyone know how to send him questions? Thanks Carla
Technical SEO | | Carla_Dawson0 -
Robots.txt
Google Webmaster Tools say our website's have low-quality pages, so we have created a robots.txt file and listed all URL’s that we want to remove from Google index. Is this enough for the solve problem?
Technical SEO | | iskq0 -
Domain Name Acquisition
Hi Anyone know how to get contact details for a domain owner beyond those provided by the likes of whois, archive etc etc ? Cheers Dan
Technical SEO | | Dan-Lawrence0 -
Quick robots.txt check
We're working on an SEO update for http://www.gear-zone.co.uk at the moment, and I was wondering if someone could take a quick look at the new robots file (http://gearzone.affinitynewmedia.com/robots.txt) to make sure we haven't missed anything? Thanks
Technical SEO | | neooptic0 -
Need Help With Robots.txt on Magento eCommerce Site
Hello, I am having difficulty getting my robots.txt file to be configured properly. I am getting error emails from Google products stating they can't view our products because they are being blocked, and this past week, in my SEO dashboard, the URL's receiving search traffic dropped by almost 40%. Is there anyone that can offer assistance on a good template robots.txt file I can use for a Magento eCommerce website? The one I am currently using was found at this site here: e-commercewebdesign.co.uk/blog/magento-seo/magento-robots-txt-seo.php - However, I am getting problems from Google now because of it. I searched and found this thread here: http://www.magentocommerce.com/wiki/multi-store_set_up/multiple_website_setup_with_different_document_roots#the_root_folder_robots.txt_file - But I felt like maybe I should get some additional help on properly configuring a robots for a Magento site. Thanks in advance for any help. Please, let me know if you need more info to provide assistance.
Technical SEO | | JerDoggMckoy0