Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How many internal links should a (sub-)page maximal have?
What would the effect of having too much be? Does google stop crawling after a number and punish more internal links?
On-Page Optimization | | brainfruit0 -
Duplicate Blog pages across different domains
Hey Moz Community, I have 3 Duplicate websites which more or less contain the same blog article ( they are copy & paste from the original website ). I am now in the process of changing my duplicate websites and I stumbled upon this problem: if I have to change the content for all the duplicate articles I have across my different domains it would be a very time consuming task and on the other hand I don't want to no index, follow the duplicate articles because I want to use them for SEO purposes. Should I only change the articles that brought significant traffic and no index, follow the rest ? What do you think ? Thanks, Anddrei
On-Page Optimization | | kiraftw0 -
How can i block the below URLs
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.? http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/ My robots.txt file is - User-agent: * Disallow: /wp-admin/
On-Page Optimization | | MasonBaker0 -
Can't get my preferred URL, how much does it matter?
Hi guys. I'm building a new site at the moment - seen a solid SEO opportunity for my work. I'm a producer engineer, specialising in mixing and mastering, so i'm creating a site for online mixing services. After a bit of keyword research I decided that "online mixing" was the best, most relevant and high volume term to go for. Ideally i'd like my home page to be www.onlinemixing.com (or something similar) but alas! It's been taken, as well as all the variations (like switching words, hypens etc) How much does this matter form an SEO point of view? E.g - For the search term "online mixing" would - www.onlinemixing-signalchain.co.uk be much worse than -www.onlinemixing.co.uk? Or am I sweating the small stuff? Any thoughts would be greatly appreciated. Isaac.
On-Page Optimization | | isaac6630 -
Is Google indexing something I can't see on my page title?
When I do a search for my businesses website (colourpages.com) it returns a listing which is great but the title tag includes Hull Colour Pages which is our old brand name - we changed to colourpages.com 2 years ago yet Hull Colour Pages is still pulling through on the title tag. I have checked the source and it's as follows: <title>Plumbers in Hull - Reviews - colourpages.com</title> I have attached a photo which shows the issue - http://imgur.com/Cva3001 Is Google indexing it from old history that is no longer visible or am I missing something? Cva3001
On-Page Optimization | | colourpages0 -
How Can I Fix Adobe Bridge Photo Galleries and Duplicate Content?
I have used the Adobe bridge program for a number of photo galleries on a remodeling site and it is showing a large amount of duplicate titles, etc. Is there an easy fix to this? anyone?
On-Page Optimization | | DaveBrown3330 -
Duplicate biographies across several domains?
Hey Gang, We've built a niche specific, manually edited legal community that is full of unique content. (While it's ultimately a directory, and that is often viewed as a bad word in these times, ours is a curated source that doesn't allow anyone and everyone to join.) We feel comfortable that it passes the sniff test post-panda/penguin, etc., and its doing rather well to date. The question we have is do we really need to create unique biographies for each of our legal members? Some of our competitors simply use the same bio information that the lawyer has on their own website and copies that to their site. The competitors I'm talking about are LARGE, well-respected, extremely successful folks, like FindLaw. Here's an example: Lawyer's website bio FindLaw website bio Both rank, etc., and the FindLaw code doesn't place any restrictions on their content regarding the bio., and it's an obvious exact match. I get that duplicate content is primarily a concern among one's own URL and the pages across a specific domain, using rel-canonical, etc., but what about two different domains that need to supply factual information that can't be altered? Is it anything we should worry ourselves with? Any tags we should insert in our code with regard to the bios? Thanks!!! Wayne
On-Page Optimization | | Wayne760 -
Multiple local addresses for one domain
Hi Mozzers! If I have a website that has a separate office in each state that they wish to rank locally for eg: "service CityA" "service CityB" etc Would it be best to have a separate subdomain for each major city (unique content on each) with the office address specific to that city? Then there would be a separate Google places listing the subdomain as the website. Or should we just use subfolders? Would Google Places accept the subfolder as the web address? thanks!
On-Page Optimization | | Netboost1