How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking drop after new website
Hi there, I have a new client who has just had a new website built (by someone else). It was quite a major change as it was 12 years old and has just been moved to Wordpress. However although they are by and large happy with the new site, they have lost a lot of their rankings in Google. The content and menu structure is apparently identical. I told them I didn't think this was unusual but I'm not sure how easy it will be to get them ranking again. Where are they likely to be starting from? Is it a case of starting from the beginning or will there be some residual ranking capability left over? Or can they expect a full recovery over time? I was going to start by looking to see if things like tagging and meta data has been filled in (I will add the site to my Moz account) but is there any way of comparing the old site with the new for SEO purposes? Thanks so much, Sarah.
Web Design | | Frog-Marketing0 -
What website changes (technical) SEOs can ignore confidently? Google's perspective!
Hi community members, I am looking after SEO at our company and there are lots of changes happening about our website; especially technical changes. It's hard for me to look after every deployment of the website like change of server location, etc. We generally agree that every change related to website must be notified by SEO to understand the ranking fluctuation and how search engines welcome them. I just wonder what technical deployments of a website I could confidently ignore to save time and give a go ahead to technical team without interrupting or waiting for my approval. Thanks
Web Design | | vtmoz1 -
Is there a best practice for how to set up an Age Restriction prompt for a website (vape equiptment)
I am working on a website that sells vape equipment and supplies and is required to have an age restriction (birthday) prompt in order to enter the site. Is there a 'best practice' way to implement this?
Web Design | | RandolphMorris0 -
Responsive design or mobile website for SEO
Because domains with a mobile version have better rankings in Google then domains with only a deskop version we're considering a mobile website. My web developer says that a m.domain.com is duplicated with the domain.com, and he recommends a responsive design. What is better for SEO positions in mobiles devices. A m.domain mobile website or a responsive design. What are influential factors?
Web Design | | remkoallertz0 -
Website subscribe form.
Hello, Im working on a clients website and I have 2 box's. One is a subscription box and the other is a newsletter sing up. Subscription box is a google feedburner where every time there is a new post, it automatically notifies the readers. Whats the best strategy to have subscribe box since its confusing for readers when you have 2 forms. Thank you for your help.
Web Design | | KentR0 -
Wordpress or custom built website?
Hello fellow mozzers, Hope you are all well. I am looking to get a website done and I am struggling to decide whether to get a custom build website site done or a website built on the WordPress platform. Would you be able to share your experiences and advice/suggestions on what you would use, and why? (pros and cons, etc.) Thanks in advance. Kind Regards
Web Design | | JonathanRolande0 -
SEO list for creating the *perfect* website
If you could build your website from scratch and have your developers do anything you want (within reason), what list of SEO requirements would you send them? Does anyone know of any good articles on the perfect SEO wish list? Happy Holidays!
Web Design | | MirandaP1 -
Best Ways to Handle a Website for Different Country Locations?
Currently our website is being redesigned into a CMS for our Headquarters in the US. We have three other locations. Germany, UK and Australia. Each Location has there own domain and hosting in there designated country. How safe will it be to upload our New Headquarters site to each of our different locations? I would like to stay uniform throughout. The content would be similar, but translated to the correct language. And some of the products would be different. Will this be a pretty easy process and will this cause any duplication issues? Are there any suggestions or MUST DO's for localization websites? The website is being designed with Wordpress and Magento as a custom CMS.
Web Design | | hfranz0