Best way to block a sub-domain from being indexed
-
Hello,
The search engines have indexed a sub-domain I did not want indexed its on
old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.
My main domain default robots.txt says :-
Sitemap: http://www.domain.com/sitemap.xml
global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: /comments/
Disallow: /? -
Hi,
CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.
To be doubly safe, you can also include a meta noindex tag on dev pages.
Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.
Hope this helps,
Kristina
-
I've never tried a method like this - FreshFireOne, did you?
-
First and foremost when you finish all this - password protect your dev instances. A url will leak out eventually and then this happens. I know it is a PIA, but it is worth it.
To remove subdomains. Go into GWT and register the subdomains as separate websites in GWT. Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files. If you cant do that, have your subdomains include a noindex meta tag on all pages. You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done. Talk to your devs. Then go into GWT and use the URL removal tool. Just leave it blank and it will remove the whole site.
Poof. Gone. You can then watch the GWT accounts. They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request." This is a good error as it confirms that that subdomain is removed.
We actually used this not on a dev site but on our www1 server that was indexed. We use a load balancer with multiple copies of the site. www1 was completing with www. Using this above did the trick.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How Can I Redirect an Old Domain to Our New Domain in .htaccess?
There is an old version of http://chesapeakeregional.com still floating around the web here: http://www.dev3.com.php53-24.dfw1-2.websitetestlink.com/component/content/category/20-our-services. Various iterations of this domain pop up when I do certain site:searches and for some queries as well (such as "Diagnostic Center of Chesapeake"). About 3 months ago the websitetestlink site had files and a fully functional navigation but now it mostly returns 404 or 500 errors. I'd like to redirect the site to our newer site, but don't believe I can do that in chesapeakeregional.com's .htaccess file. Is that so and would I need access to the websitetestlink .htaccess to forward the domain? Note* I (nor anyone else in our organization) has the login for the old site. The new site went live about 9 months before I arrived at the organization and I've been slowly putting the pieces together since arriving.
Intermediate & Advanced SEO | | smpomoryCRH0 -
Domain Name
Hello everyone Please advice what to do in a situation when searching for a domain: www.domain.com google is recommending domain.org ? when these are completely 2 different sites? Does it has to do with trust rank? Please advice.
Intermediate & Advanced SEO | | FusionMediaLimited0 -
Sitemaps during a migration - which is the best way of dealing with them?
Many SEOs I know simply upload the new sitemap once the new site is launched - some keep the old site's URLs on the new sitemap (for a while) to facilitate the migration - others upload both the old and the new website together, to support the migration. Which is the best way to proceed? Thanks, Luke
Intermediate & Advanced SEO | | McTaggart0 -
What is the best strategy for linking to sub category pages?
My site is set up like this (i have x6 categories and all are similar) Home Page - Category - sub category - X4 detail pages My category page provides a summary/introduction of the subject, my sub category page is the "money page" with ability to quote & buy - my detail pages provide supporting material. What is the best internal linking strategy between these pages? (in addition, in one category i have x6 sub categories but only one of them is a "money page", should i be linking all of these pages back to the money page?) Thanks Ash
Intermediate & Advanced SEO | | AshShep10 -
Domain Forwarding
I have a client who previously registered 20 unique domain names that tied to their company name and services. They use all of these domains to forward to their main website to try and capture additional traffic. Would you suggest that we remove all of the domains by 301 redirecting them all the the main website? I am trying to find a good article that shows the implications on SEO by using many domains that forward to a main website. Any suggestions would be greatly appreciated.
Intermediate & Advanced SEO | | Prager0 -
What is the best way to rank well in two countries simultaneously with only one CCTLD
I have a .co.nz website and would like to rank on .com.au without setting up a new country specific website for .com.au. What is the best way to do this ?
Intermediate & Advanced SEO | | SteveK640 -
Moving a database from a sub folder to a new domain
Hi Mozzers, I have a very popular property database on a subfolder of my main website. It accounts for about 2 thirds of my overall traffic. I'm moving the database to a new server, so my choices are now a sub domain or completely new domain. If I move my database to a new domain, will my main site lose SEO because of how popular the database is? Cheers Matt
Intermediate & Advanced SEO | | Horizon0 -
Best way to get the keyword ranking at the top
I am working on site for around six months now.
Intermediate & Advanced SEO | | ray2810
I have done social bookmarking submission, directory submission, blog comments, forum submissions etc. Is there anything else i can do to make the rank go higher. nothing is working correctly.0