Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I analysis my domain (http://yourdigibuddy.com/) on moz domain analysis. But there is not data show and show message "No data here". What should I do?
_ I analysis my domain (http://yourdigibuddy.com/) on Moz domain analysis. But there is not data show and show the message "No data here". What should I do?_
On-Page Optimization | | MelissaDooley0 -
Can I have schema.org links as relative on my site? Getting an html validation error.
I'm getting an html validation error on relative schema.org links "Bad value //schema.org/Organization for attribute itemtype on element div: The string //schema.org/Organization is not an absolute URL." This is my code for https site: <code class="input">e itemtype="//schema.org/Organization"><a itemprop="url" class="navbar-brand" …<="" code=""></a></code>
On-Page Optimization | | RoxBrock0 -
How updating a post can influence seo
I have just read this great post from moz blog: http://moz.com/blog/google-fresh-factor But I haven't found nothing about updating post date. If I edit an old great post (now ranked 2nd after several months in 1st serp position) does it be better if I also update the date of the post in the wordpress post edit page? Thank you very much and sorry for my poor english. Bye, Dario.
On-Page Optimization | | Italianseolover0 -
Same UI, different domain
Hello there. I would like to ask for your opinions. I have DomainA.com/blog and want to move /blog to its own domain. As of now, the blog section has the exact same UI as the rest of the site, its just a section on our site. Will it have any negative impact with respect to SEO if DomainA.com and DomainB.com have the exact same UI and are interlinking each other? Thanks in advanced.
On-Page Optimization | | mike_sif0 -
Where Can I Find Freelance Writer? Content free from Copyscape
Hi, Anyone know where i can get freelance worker that writes
On-Page Optimization | | chanel27
about 500 to 600 words original article that is free from Copyscape? And also how much does it cost in general?0 -
What does this mean on first step up setting up a campaign? "Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here."
I am BRAND new to this, and setting up my first campaign. I choose subdomain, and entered www.pdsaz.com. This is the message I receive: We have detected that the domain www.pdsaz.com and the domain pdsaz.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here.
On-Page Optimization | | cschwartzel0 -
How do you block development servers with robots.txt?
When we create client websites the urls are client.oursite.com. Google is indexing theses sites and attaching to our domain. How can we stop it with robots.txt? I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy. Thanks, TR
On-Page Optimization | | DisMedia0 -
Www1 and www domain
hi, I have a client who has an e-commerce business. My client does not want to fill the pages with too much content and has set up a www1 version with the same domain-name as the www. The plan is to create a lot of content and push www1 in ranking and then sending users (via links) to the www for ordering. Although there will be no duplicate content published on www and www1 this seems like an odd strategy, especially since the www already has a good page rank, and I'm not sure about how engines view a www.domain.com and www1domain.com situation even with unique content in each. Any thoughts?
On-Page Optimization | | vibelingo0