Can Robots.txt on Root Domain override a Robots.txt on a Sub Domain?
-
We currently have beta sites on sub-domains of our own domain. We have had issues where people forget to change the Robots.txt and these non-relevant beta sites get indexed by search engines (nightmare).
We are going to move all of these beta sites to a new domain that we disallow all in the root of the domain.
If we put fully configured Robots.txt on these sub-domains (that are ready to go live and open for crawling by the search engines) is there a way for the Robots.txt in the root domain to override the Robots.txt in these sub-domains?
Apologies if this is unclear. I know we can handle this relatively easy by changing the Robots.txt in the sub-domain on going live but due to a few instances where people have forgotten I want to reduce the chance of human error!
Cheers,
Dave.
-
Hi Dave. A workflow checklist should really help with this as well. There are probably a few other items you'll catch by meeting with the others involved and getting everyone on the same page. Cheers!
-
Dave, I had exactly the same issue a month ago with being indexed on subdomains but was able to modify robots.txt in root domain swiftly enough to avoid real damage. Your main root robots.txt can override the subdomains. Simply disallow the subdomain in your robots file disallow: /subdomain url/robots.txt OR, if im not mistaken simply remove it altogether from the subdomain
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Optimise root level page for secondary keyword or try to make second level rank?
My Main keyword ranks no where in the top 100 for any pages in my site. for / I am ranked at number 10 for a great secondary keyword. Really want to get this secondary keywordranking on /secondary-keyword.html , should I focus my main page only on my primary keyord and remove any references to "Secondary Keyword"? What I feel here is that I would be sacrificing my rank for "Secondary Keyword"
On-Page Optimization | | Adamzoz0 -
How we can rank two keywords
Here is my example. I developed a property inspection app, where the keyword I try to rank is property inspection app, and property inspection software. Property Inspection Software got slightly more traffic than app. I would like to rank both of this keyword high for my main website, but there is only one title, one H1 tag I can use to have keyword, where if I use "App" there is no way to have "Software" on the title, etc. How can I deal with such a case? Any help will be appreciate.
On-Page Optimization | | terrysun0 -
Can any one give some advise on how to work out which back links are bad
we are trying to clean up our site and believe we have bad back links can any one tell me how to work out which ones to disavow
On-Page Optimization | | CostumeD0 -
Transfer Blog From External Domain
Hi All I am in stages of transferring a blog. We currently have partial manual penalty for unnatural links, and we received sample links of our own blog, which happened to be originally setup on another domain. e.g. example.co.uk and the blogs hosted on exampleblog.co.uk - the only way these were linked were through anchor text in each and every blog - so it's clear why this has been picked up. We are now going to transfer these across to our actual domain. Around 60% of the blogs had no meat, so we have pruned through these. What is the best solution; Redirecting ONLY the good blogs via 301 Redirect and 404 pages with no meat OR Redirecting ONLY the good blogs via 301 Redirect the others to Example.co.uk/Blogs Thanks Tom
On-Page Optimization | | TomPryor830 -
New jobboard: Can redirecting folder (site.com/jobboard) to subdomain (jobboard.site.com) hurt SEO?
Hi there, I'm planning to implement a jobboard on my website which needs to be installed on a subdomain (jobboard.site.com) but I'd really like to use site.com/jobboard for promoting this jobboard (jobboard collects external industry jobs). Are there any possible disadvantages when I set up a 301 redirect from jobboard.site.com to site.com/jobboard? Also: What if I want to move this jobboard to a unique domain one day (e.g. jobboard-industry-xy.com), Would that be tricky (as I'd basically have to redirect the folder-to-subdomain redirect to an external domain and therefore get a folder-to-subdomain-to-external-domain redirect...)? Cheers, Thomas
On-Page Optimization | | stl990 -
Is .PW domain is good for SEO?
I want to register .PW domain which has recently got live to register. I am in doubt should it is good for SEO or not.
On-Page Optimization | | semmediapvtltd0 -
Can Rankings in Google differ so much from computer to computer.
I was telling my friend via facebook to go on my website, I told him to search 'nightlife forum' in google. To which, I believed it was 11th, top of second page. On his computer, its currently ranking at 1st place is it possible to have a difference of 10 places? even though he lives in the same city as me. Would be good to see what it ranks on your computers too google "nightlife forum" look for www.talknightlife.co.uk (don't get confused with the .COM one out there) Cheers Guys
On-Page Optimization | | Lukescotty0 -
301 redirects from several sub-pages to one sub-page
Hi! I have 14 sub-pages i deleted earlier today. But ofcourse Google can still find them, and gives everyone that gives them a go a 404 error. I have come to the understading that this wil hurt the rest of my site, at least as long as Google have them indexed. These sub-pages lies in 3 different folders, and i want to redirect them to a sub-page in a folder number 4. I have already an htaccess file, but i just simply cant get it to work! It is the same file as i use for redirecting trafic from mydomain.no to www.mydomain.no, and i have tried every kind of variation i can think of with the sub-pages. Has anyone perhaps had the same problem before, or for any other reason has the solution, and can help me with how to compose the htaccess file? 🙂 You have to excuse me if i'm using the wrong terms, missing something i should have seen under water while wearing a blindfold, or i am misspelling anything. I am neither very experienced with anything surrounding seo or anything else that has with internet to do, nor am i from an englishspeaking country. Hope someone here can light up my path 🙂 Thats at least something you can say in norwegian...
On-Page Optimization | | MarieA1