How to prevent development website subdomain from being indexed?
-
Hello awesome MOZ Community!
Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.
An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.
The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.
My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?
Thanks!
-
Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.
You can find information on setting this up here.
-
Thanks for your prompt feedback Oleg!
Your proposed action is exactly what we're implementing.
Randy Holland, Sprout Digital
-
So....
- If the dev site has not been indexed yet, you can block crawlers via robots.txt
- If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again
As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Website redesign- change of server . What to do with old site? Keep for a while or delete right away?
Hey Mozzers, Two days ago, we redesigned our website and changed the server at the same time to get faster loading times. Here is what we have done. The old site was hosted on ipage, new site with a new design hosted on UPCLOUD. We changed the A record to the new server, uploaded a new site, submitted a new sitemap to Google Search console, 301 redirected all old URLs to new ones, most have changed a bit. Old URLs were ending with " .html "the new ones do not have that at the end. Submitted AMP pages to Google as well. Now here is my question. Should we delete the old site completely from ipage or should we keep it for a while? Google has indexed the new URLs that were created with the redesign, these URLs did not exist on the old site. But it still shows most of the old URLs on SERPs (these are URLs that have been 301 redirected to a new equivalent page) I understand 2 days is not very long for Google to get everything right, but I am not sure what we should do with the old site? Keep it or get rid of it to help Google index the new one only. FYI every single old URL that appears on Google search when clicked on will take you to the right place, we made sure there are no 404s at all. As this is very important to our business and we get most of it from Google I want to make sure we do it right for SEO purposes. The agency that designed the site did not really know the answer to that question, as they do not have SEO specialists. Please help, any input you might have will be greatly appreciated.
Web Design | | Davit19850 -
When rel canonical tag used, which page does Google considers for ranking and indexing? A/B test scenario!
Hi Moz community, We have redesigned our website and launched for A/B testing using canonical tags from old website to new website pages, so there will be no duplicate content issues and new website will be shown to the half of the website visitors successfully to calculate the metrics. However I wonder how actually Google considers it? Which pages Google will crawl and index to consider for ranking? Please share your views on this for better optimisation. Thanks
Web Design | | vtmoz0 -
Website Traffic Drop After Switch
Hey guys, two weeks ago we switched over to the new website, however we've experienced about a 20% drop in Google Organic traffic and it does not appear we are getting indexed correctly by Google. On search results it's not using the meta description and the links still point to the old pages. Robots is fine We are using the same IP address Re-directs are in for the pages in question. Sitemap was submitted to Google in Webmaster tools What else do we need to do?
Web Design | | ScottOlson0 -
Looking for a developer with Volusion experience
We are looking for experienced developer that specializes in Volusion sites. Thanks in advance! Tony 714-744-1926
Web Design | | OCFurniture0 -
How would restructuring the navigation of my website affect my rankings?
I want to restructure the navigation of my website for a few reasons: 1. It isn't intuitive/clear to the user 2. It is way too big, it has too many links and thus causes the number of links on many pages to be >100. 3. I want to get rid of file extensions as part of the URLs (.html, .php) 4. I want to achieve a "tree"-like navigation system, with categories, subcategories and so on. In the process of cleaning up my website, I had to 301 redirect a lot of duplicate pages, fix broken links, etc. I have a lot of 301 redirects already, and in the process of restructuring the navigation of my website I know I'm going to get more. Will the addition of new 301 redirects have an effect on my rankings? (I'm basically going to be changing all of the URLs) What kind of SEO effect will restructuring the navigation at the top of the page (reducing the # of links on the main menu) have on my site? What is the best strategy to implement in this situation?
Web Design | | deuce1s0 -
Need an SEO pro to build us a new fine art website
Hey Seomoz friends My friends and I started buying and selling art 5 years ago. We did it all on ebay. Two years ago I decided to try an attempt at building us a website to further our business outside of ebay. www.originalartbroker.com I used homestead.com software aka intuit to do this. I was able to get a lot of first page results in seo after a ton of work and willing to do a lot more. I'm sure the homestead site slowed my efforts down. I need more power 🙂 What we do. We sell fine art. We are trying to aspire to the leading seller of fine contemporary art online. We will do whatever it takes to get there. Our issue is not our product. Our issue is getting traffic to the site. I only have 175-200 visitors a day. I need better results for the keywords of our artists and inquires that surround them. We have a bounce rate of 70%, average time on site of 1 minute 50 seconds, and a slow loading site that is almost impossible to create individual pages for pieces (mgmt nightmare) Outside of our site we have blogs, squidoo pages, facebook account with 2500 fans, twitter page with 2k followers, youtube channel, and a blog on the site for new arrivals. We have a 200-250 piece on our site at any given time. We add probably ten pieces on average a week. We don't need any kind of ecommerce management software as most of our sales are done over the phone being that it is higher end art. I need a site that out performs my competition. I need to be in the top three when someone types in "leroy neiman" "peter max" and so on. I, with limited knowledge, need to be able to use the software everyday to upload new art as it comes in. I want every piece to have its own page so that I can also add pieces to a google merchant account. When I add new pieces under a certain artists and it creates its own page i want it to create h tags and an url extension for each product as I add them based off the discription and artists name. I would like some sort of blog integration to post our new arrivals as we get them. I would like some sort of customer capture. I am thinking something along the lines of them prociding email, name, and zip code to see the prices on the site. You are the seo pro so you know what it takes. I would like to know what a solution would cost to get us on the ground with more seo power. A site with speed that is easily indexed. It doesn't need to have a lot of bells and whistles. Please look at my site and let me know what you think. You can get an idea from that of what we are doing. Please give me an idea of what you can do and what it would cost. Thank you
Web Design | | forecastedinvestments0 -
How does the "first link" rule work with the "reasonable surfer patent" when it comes to the main navigation for a website?
In trying to figure out navigation for a new website, I am struggling with the first link rule vs. the reasonable surfer patent where the first link rule implies that Google "counts" the first link to a page including navigation, and the reasonable surfer patent that implies that navigation links carry less weight than body copy links. What is the best solution for creating main navigation so that it doesn't take away from the body copy links?
Web Design | | cindyt-170380 -
Old links in Google, new website affecting SEO?
Hi Guys, I have launched my website in october and it has already been indexed by google. Now I'm going to launch my redesign which comes with a new structure, content, links, etc. So the question is, do I have to resubmit my website to google to get rid of old links? Onsite Explorer shows links to my forum which has been spammed with p* stuff which has been already indexed as well. The forum is off now. I want to use SEOmoz to track my new website but I guess this could be a hard thing as old links etc will be shown as well. Is there any tool to let Google know about my changes? Does it affect my SEO in any way? Thank you for your help. Nick
Web Design | | NickITW0