Getting subdomains unindexed
-
If i turn an application off displaying a 503 error will that get my site unindexed from search engines?
-
Subdomains can be verified as their own site in GWT. Verify the subdomain in GWT, then put a robots.txt on that subdomain excluding the entire subdomain, then request removal in GWT of that entire subdomain. I've had to remove staging and dev sites a couple of times myself.
A couple of things I've found useful in this situation is to make the robots.txt files for both the dev and live sites read only, so you don't accidentally overwrite one with the other when pushing a site live. You can also sign up for a free tool like Pole Position's Code Monitor that will look at the code of a page (including your robots.txt url) once a day and email you if there are any changes so you can fix the file then go hunt down whoever changed the file.
-
GWT was the first placed i checked unfortunately you can only remove directories or pages. I need entire subdomained sites to be removed (in fact they shouldn't of been indexed in the first place).
We use subdomains for our development testing environment when creating client sites and once the site is approved we push it live replacing the old site. Somehow these testing sites are getting indexed and it may pose a threat to duplicate content on different domains. So i am trying to find a solution to get the subdomains (100's of them) unindexed.
I understand a 301 redirect is best but that isn't really applicable since these test sites still need to be reached by clients.
-
With a robots.txt blocking it, you can then go into Google Webmaster Tools and request removal of that particular page or folder from Google's index.
-
No index tag on it works, and putting up a robots.txt that disallows everyone should work as well.
-
Thanks for the quick reply, i will have to try that. Essentially i am trying to get the site un-indexed but i wasn't sure if a 503 would do the trick.
-
Eventually, but that's the code Google recommends to return when your site is having downtime, so I would expect them to be more lenient towards not removing things right away. I wouldn't expect it to be as efficient as returning a 404 or a 410.
The best way to get content de-indexed is to return a page with a meta noindex tag on it, if you're really keen on getting it removed immediately.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why images are not getting indexed and showing in Google webmaster
Hi, I would like to ask why our website images not indexing in Google. I have shared the following screenshot of the search console. https://www.screencast.com/t/yKoCBT6Q8Upw Last week (Friday 14 Sept 2018) it was showing 23.5K out 31K were submitted and indexed by Google. But now, it is showing only 1K 😞 Can you please let me know why might this happen, why images are not getting indexed and showing in Google webmaster.
Technical SEO | | 21centuryweb0 -
Do YouTube videos in iFrames get crawled?
There seems to be quite a few articles out there that say iframes cause problems with organic search and that the various bots can't/won't crawl them. Most of the articles are a few years old (including Moz's video sitemap article). I'm wondering if this is still the case with YouTube/Vimeo/etc videos, all of which only offer iFrames as an embed option. I have a hard time believing that a Google property (YT) would offer an embed option that it's own bot couldn't crawl. However, let me know if that is in fact the case. Thanks! Jim
Technical SEO | | DigitalAnarchy0 -
Test site got indexed in Google - What's the best way of getting the pages removed from the SERP's?
Hi Mozzers, I'd like your feedback on the following: the test/development domain where our sitebuilder works on got indexed, despite all warnings and advice. The content on these pages is in active use by our new site. Thus to prevent duplicate content penalties we have put a noindex in our robots.txt. However off course the pages are currently visible in the SERP's. What's the best way of dealing with this? I did not find related questions although I think this is a mistake that is often made. Perhaps the answer will also be relevant for others beside me. Thank you in advance, greetings, Folko
Technical SEO | | Yarden_Uitvaartorganisatie0 -
How can I get the most out of uploading a print magazine to my client's website?
Hi Mozers, My client is just about to launch a print magazine for her watch business. There is so much valuable content in the magazine and we want to feature it on the website both for SEO purposes and also for those who prefer to read articles online instead of reading a physical magazine. My question is: what is the best method of displaying the magazine to get the most from search rankings and also to capitalise on the beautiful imagery from the magazine. The best option that I can think of is to upload the magazine as a flipbook and create a separate page on the website to display each article so that search engine crawlers can index the content. I do understand that this could be problematic if users are only spending time reading the flipbook and not so much time on the article pages. Do you guys have any suggestions about how to get the most out of this opportunity for my client? THANK YOU IN ADVANCE. Meaghan
Technical SEO | | StoryScout0 -
No index on subdomains
Hi, We have a subdomain that is appearing in the search results - I want to hide this as it looks really bad. If I were to add the no index tag to the sub domain would URL would this affect the whole domain or just that sub domain? The main domain is vitally important - it is just that sub domain I need to hide. Many thanks
Technical SEO | | Creditsafe0 -
Automatic subdomain redirection programmed over a given period?
Hi all, I have a technical problem that is quite beyond my "expertise", and I'm wondering if someone can help me to find a way to handle this. My website automatically changes appearance and some content as well twice a year (summer and winter), and we also change the sub domain, like this: September to may > www.nameofmysite.com and june to august > summer.nameofmysite.com I know it's a bit tricky, and we're doing this with a 303 redirection and will also provide a possibility (via a button) to switch beetwen "seasons". To sum up, during the "summer" I've got: http://summer.nameofmysite.com/webzine/news/name-of-my-page.html and during the "winter": http://www.nameofmysite.com/webzine/news/name-of-my-page.html As you can see, it's the same content, with a temporary redirection 303. Thanks you very much in advance! Simon
Technical SEO | | Simon-0 -
Domains and subdomains
When I started a campaign for my message, I got the message: "We have detected that the domain www.vamospaella.com and the domain vamospaella.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here." I wasn't sure whether I had said it was a subdomain when in fact it was a domain (or the other way round), so I started another campaign for the same website using the other option and the message didn't come up. However, I still don't understand what you meant by this and whether it's an issue. When I search for my website in Google, it shows as vamospaella.com when other websites come up as www. and then their domain name. If it is a problem, is it to do with my hosting package and how it's set up or is it to do with my local site on my computer? I did ring my web host, 1&1, but they said they couldn't see a problem. Please can you let me know how I can resolve this as my ranking is still quite low in Google and I'm not sure why. If it is because of "twin domains", then will Google see my content as duplicated and keep me low in their rankings? I'm new to SEO and not a website novice, so please answer in lay terms! Thanks Melissa
Technical SEO | | melissa10 -
If you add a no follow to a time sensitive link, will it get picked up as broken link 404 in WMT report?
We have a client who publishes deals that are time sensitive. Links to the deals expire and so Google's crawlers are picking them up and finding a 404 If I no follow them, will the 404's still get picked up and reported in WMT? The same question applies to SEOMoz Pro.
Technical SEO | | Red_Mud_Rookie0