How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Business Services via Index Page or Dedicated Pages?
We're in the process of building a new website for our Business (B2B Event Services) and we've hit a design snag. Our designer wants to combine all of our business services information (there are six service lines) into a single Index Page titled "Services". Beyond this creating a needlessly long page to scroll through, I'm worried this will negatively impact our ability to SEO for each service line, as we wouldn't have any intention of letting people visit the individual pages from within the site. Our current site features individual pages which, in my opinion, is how we should build our new site. That being said, I'm completely open to any ideas that will further enhance usability and searchability. Any clarification would be greatly appreciated!
Web Design | | SHWorldwide1 -
App indexing with several subdomains
Hi, We are currently in the process of getting our app developed. From an SEO point of view I want the app to get indexed and am currently trying to get a step by step guide for it. I've come across this helpful guide: https://moz.com/blog/how-to-get-your-app-content-indexed-by-google but need to go a bit deeper now. Questions I have is related to the "apple-app-site-association" file As far as I could find out the JSON code of that file has the following syntax: {_ “applinks”: {__ “apps”: [],__ “details”: [__ {__ “appID”: “9JA89QQLNQ.com.my.bundle.id“,__ “paths”: [ “/myPath1”, “/myPath2/*” ]__ },__ ]__ }_ } My question is related to the key "appID" I'm wondering how the entry would look like if I have 10 country subdomains and would like to make sure that the deep link clicked in the browser is going to the right subdomain page within the app. Current website structure:
Web Design | | FashionLux
uk.example.com/english-path/
de.example.com/german-path/ Many Thanks in advance0 -
Making a website menu + structure + hierarchies + kw research
When making a new website structure I assume we all think about SEO but at the same time as we can't forget UX.
Web Design | | ceranoktan
How much of your KW research would you implement in menu \ website structure?
What do you think is important to think about when making a website structure? Thank you in advance, BR,
Ceran0 -
Best Practices for home page design for ecommerce website
I know this question is not directly related to SEO, but I figured I have been getting some good help from this forum, so why not? The website is www.vrtack.com. I am looking to redesign the home page. It is an ecommerce website selling equestrian clothing and leather goods. My goals are: 1. Reduce the very high bounce rate and drop-off rate. 2. Fine tune the relevancy of the website towards a handful of keyword phrases. 3. Engage the visitor to create better click-through and to increase the average time spent on the page/site. 4. Page Loading time is of importance. It has to load quickly. I would love to hear some specific suggestions, examples, best practices.
Web Design | | amitramani0 -
Need help in website URL Structure
I have been working on a brand new website currently it is live but I have disallow Googlebots temporarily as I dint want any negative impact. The business of the site is to generate leads , they install and sell Stairlifts and used Stairlifts. There are two main categories New Stairlifts and Reconditioned Stairlifts Currently the URL for new Stairlifts is : http://willowstairlifts.co.uk/stairlifts/ and for Reconditioned Stairlifts is: http://willowstairlifts.co.uk/reconditioned-stairlifts/ My concerns are that the word Stairlifts is mentioned twice in the urls so is it going to have a negative impact or panda penalty? I am thinking of changing them to http://willowstairlifts.co.uk/new/ and the product pages to display as http://willowstairlifts.co.uk/new/brooks/ Currently its http://willowstairlifts.co.uk/stairlifts/brooks/ Same with reconditioned Stairlifts I like to change it to : http://willowstairlifts.co.uk/reconditioned Also its product pages to http://willowstairlifts.co.uk/reconditioned/brooks/ As currently its http://willowstairlifts.co.uk/reconditioned-stairlifts/brooks/ Thanks
Web Design | | conversiontactics0 -
Changing URL's for a website redesign
Hi folks, We're redesigning our website and looking for some advice on how changing our URL's would affected our rankings. If the page URLs are changing how can we carry out redirects to avoid losing any SEO rank? Thanks, Ross
Web Design | | Will_Craig0 -
SEO while designing the website and continuous SEO
What's the difference between SEO while designing/developing a website (ie, setting up a website so it is crawled by search engines) and the SEO that people talk about that needs to be viewed analyzed and changed all the time? I don't have a ton of money to spend on SEO right now but I do want to make sure my website is set up by an SEO expert (if possible) so when I do have money to spend on SEOthe website is set up properly to work with more advanced forms of SEO. I guess I don't understand where you can draw the line (if forced due to money constraints) between SEO in the beginning and continued SEO. What do I need to look for with SEO and the design of my website. How do I get some type of SEO without breaking the bank??
Web Design | | CapitolShine0 -
Where should I spend Money on my website?
My website is www.capitolshine.com what do you think? Where should I spend money to enhance SEO search results? I do have a limited budget but the company is growing quickly and I might have more funds to invest in a few months. Where should I spend money now (less than $500 per month) and where should I spend money in the future? I am afraid the person who coded my website wasn't well versed on SEO. There also might be coding errors. I'm trying to work through the errors myself via the repots from SEOmoz.
Web Design | | CapitolShine0