Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. Digital Marketing
    3. Web Design
    4. How to prevent development website subdomain from being indexed?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    How to prevent development website subdomain from being indexed?

    Web Design
    3
    4
    4145
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • SproutDigital
      SproutDigital Subscriber last edited by

      Hello awesome MOZ Community!

      Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.

      An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.

      The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.

      My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?

      Thanks!

      1 Reply Last reply Reply Quote 0
      • Chris_Hickman
        Chris_Hickman last edited by

        Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.

        You can find information on setting this up here.

        1 Reply Last reply Reply Quote 1
        • SproutDigital
          SproutDigital Subscriber @OlegKorneitchouk last edited by

          Thanks for your prompt feedback Oleg!

          Your proposed action is exactly what we're implementing.

          Randy Holland, Sprout Digital

          1 Reply Last reply Reply Quote 0
          • OlegKorneitchouk
            OlegKorneitchouk last edited by

            So....

            1. If the dev site has not been indexed yet, you can block crawlers via robots.txt
            2. If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again

            As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.

            SproutDigital 1 Reply Last reply Reply Quote 2
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • calvinkj

              Does changing content and design of the website gonna affect my all the backlinks i have made till now

              i have been working on my link profile for a month now, after learning about 5 step moz methodology i have decided that i would like to change all of the content of my site and taylor it to what my customers need, am i gonna loose all the domain authority if make changes? if it gonna affect, hows that  gonna come out

              Web Design | | calvinkj
              0
            • Stew222

              Is The HREF Link "Title" Tag Needed on Mobile Websites?

              Hello To Those Who Are Wiser Than I, I am wondering if the href link "title" tag is needed, or serves any purpose, on mobile websites?  Also, does it effect SEO in any way? I ask because generally the href link title tag provides more information to the user when they scroll their mouse over the link - but this action does not happen on mobile!  Users have no mouse and thus no extra information would be displayed. I'm really wondering if it still matters for SEO purposes on mobile though. -The UnEnlightened

              Web Design | | Stew222
              0
            • Stuart26

              Mergers & Acquisitions - Website Transition Good practice

              Hi everyone, I was wondering if anyone has come across good practice for maintaining websites after a merger or acquisition where there needs to be an association between two websites of the two companies involved. For an acquisition, I'm considering moving the acquired company to a sub domain of the parent company e.g. aquiredcompany.parentcompany.com. On both websites there wmay be a prominant popup so visitors can switch between the websites if they have visited the incorrect one. One worry I have is the acquired company has some good rankings, which I want to keep. I will of course manage the process through 301 redirects. But I was wondering if anyone has any thoughts on this approach or can suggest any better solutions. Thanks in advance, Stuart

              Web Design | | Stuart26
              0
            • Small_Business_SEO

              Do I need to 301 redirect www.domain.com/index.html to www.domain.com/ ?

              So, interestingly enough, the Moz crawler picked up my index.html file (homepage) and reported duplicate content, of course. But, Google hasn't seemed to index the www.domain.com/index.html version of my homepage, just the www.domain.com version. However, it looks like I do have links going specifically to www.domain.com/index.html and I want to make sure those are getting counted towards my overall domain strength. Is it necessary to 301 redirect in the scenario described above?

              Web Design | | Small_Business_SEO
              0
            • TotalMarketExposure

              Subdomains For Real Estate Website

              I am currently working on a proposal for a clients Wordpress website development which includes ongoing SEO after the website is developed. I have looked into a number of options and the one that seems the most cost effective involves using subdomains for the individual listings pages. What I want: clientsdomain.com/listings/idxnumber/ What I can get for a decent price: listings.clientsdomain.com/idxnumber/ So the majority of the website will actually exist on a subdomain because the IDX API will automatically populate pages for all of the MLS listings in the area (hundreds or thousands). Meanwhile the domain itself will have all the neighborhood pages and other optimized content, blogs and whatnot. My concern is that dividing the website like this will have negative effects on SEO. There wont be duplicate content across subdomain and main domain, but they will share a lot of links back and forth. I haven't found any recent sources on the topic. Almost everything I have found says that dividing a website in this manor is bad for SEO, but these articles are often many years old. Does anyone know of a Wordpress plugin/IDX company that can provide a solution that doesn't use a subdomain and actually just lists each MLS page within a directory? I am open to using another platform, I am just most familiar with Wordpress. Will using a subdomain in the ways mentioned above have a profound negative effect on SEO? Thank you for your time in responding, I greatly appreciate it.

              Web Design | | TotalMarketExposure
              0
            • edlondon

              Is it possible to redirect the main www. domain - but keep a subdomain active?

              Hi Mozzers, Quick question, which I hope one of you can answer... Let's say I have a website (i) www.example.com and on that a subdomain exists, (ii) subdomain.example.com. Let's say I want to change my main domain from www.example.com to www.newwebsite.com.  I'd 301 all content, use GWT to notify Google of a change of address etc etc. Having done that, is it still possible to keep the original subdomain active? So, even though www.example.com has been redirected / transferred to www.newwebsite.com, subdomain.example.com would still exist. If that is possible, what is the implication for Domain Authority?  On the one hand, I have transferred the main site (so DA from that will transfer to the new site); but part of that root domain is still active. Make sense?  Any answers? Thanks everyone...

              Web Design | | edlondon
              0
            • 1SMG

              3 Brands, 3 Services, 3 Different Websites Right?

              My client was told that having 1 website for 3 different brands/services is better than having 3 websites. I need your help to prove my value to a new client.  This client has worked with Reach Local on PPC for some time and when they first got started the Reach Local Markering Consultant told this cleint that they needed to have one site for better SEO purposes. The client was told that Google ranks websites higher if they have more paid traffic going to them.  I've been doing this for long enough to realize this does not help ranking, at least not enough to make a difference. Keep in mind this is for 3 different companies.  One company does plumbing, another electrical and the last one does air conditioning.  They also have  4 locations but only two locations have mutliple services opperating out of them.  I understand these 2 location will not have there own Google+ Local / Places listing.  Using the same address for 2 different business and expecting a first page ranking is just not possible. Right now when you visit the clients website you see a logo that rotates with a banner section that follows the logo rotation.  First you see the AC Company and then the Plumbing etc.  I see this as confusing to the end user and it is more work to get it ranked for SEO.  I recommended that we build 3 speerate websites for each service and just list out all the addresses that the company services on the contact page.  I would also design inside the footer links to the other services for branding purposes. Please share your thoughts on how you would handle this if you were doing the SEO for your own 3 different business services. I really appreicate any input/insight to this.  Thank you so much in advance!!!!

              Web Design | | 1SMG
              0
            • awalker84

              Need to rebuild client's flash website

              I am working with their web designer and need to figure out a way to rebuild their site which is currently all in flash. I was wondering if there was a way to do this without spending a ton of time in completely re-doing the site from scratch.

              Web Design | | awalker84
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.