Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Best way to block a sub-domain from being indexed

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Best way to block a sub-domain from being indexed

    Intermediate & Advanced SEO
    3
    4
    2523
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • JohnW-UK
      JohnW-UK last edited by

      Hello,

      The search engines have indexed a sub-domain I did not want indexed its on

      old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them.

      My main domain default robots.txt says :-

      Sitemap: http://www.domain.com/sitemap.xml

      global

      User-agent: *
      Disallow: /cgi-bin/
      Disallow: /wp-admin/
      Disallow: /wp-includes/
      Disallow: /wp-content/plugins/
      Disallow: /wp-content/cache/
      Disallow: /wp-content/themes/
      Disallow: /trackback/
      Disallow: /feed/
      Disallow: /comments/
      Disallow: /category//
      Disallow: */trackback/
      Disallow: */feed/
      Disallow: /comments/
      Disallow: /
      ?

      1 Reply Last reply Reply Quote 0
      • KristinaKledzik
        KristinaKledzik last edited by

        Hi,

        CleverPhD has some interesting ideas with robots.txt and Google Webmaster Tools, but simply password protecting all dev pages should keep pages out of Google's index. There's no best practice here, since a password wall will keep Googlebot out on its own.

        To be doubly safe, you can also include a meta noindex tag on dev pages.

        Keep in mind that once a page is in Google's index, it's going to take awhile for it to leave (unless you use CleverPhD's method). But, having a blank page in Google's index really isn't all that bad. It's there, but it won't rank for much.

        Hope this helps,

        Kristina

        1 Reply Last reply Reply Quote 0
        • KristinaKledzik
          KristinaKledzik @CleverPhD last edited by

          I've never tried a method like this - FreshFireOne, did you?

          1 Reply Last reply Reply Quote 0
          • CleverPhD
            CleverPhD last edited by

            First and foremost when you finish all this - password protect your dev instances.  A url will leak out eventually and then this happens.  I know it is a PIA, but it is worth it.

            To remove subdomains.  Go into GWT and register the subdomains as separate websites in GWT.  Create a robots.txt for each subdomain (not the one you mention, you need a robots that is specific to that subdomain that disallows all files.  If you cant do that, have your subdomains include a noindex meta tag on all pages.   You have to be careful with this as you do not want to push out your dev. robots.txt or the noindex meta tags to your production server, but it can be done.  Talk to your devs.    Then go into GWT and use the URL removal tool.  Just leave it blank and it will remove the whole site.

            Poof.  Gone.  You can then watch the GWT accounts.  They will show errors for the dev site like "Severe health issues are found on your site - Some important page has been removed by request."  This is a good error as it confirms that that subdomain is removed.

            We actually used this not on a dev site but on our www1 server that was indexed.  We use a load balancer with multiple copies of the site.  www1 was completing with www.   Using this above did the trick.

            KristinaKledzik 1 Reply Last reply Reply Quote 1
            • 1 / 1
            • First post
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • netcomsia

              block primary . xxx domain with disavow tool

              search results spam spam score disavow seo audit

              Hi friends I discovered spam url attack on top sites with good google positions for specific keyword. Can I block primary domain like .xxx with disavow tool? There is hundreds of different domains but primary domain is always the same. for example like this domain:xxx? Thanks

              Intermediate & Advanced SEO | | netcomsia
              0
            • vtmoz

              What are best page titles for sub-domain pages?

              Hi Moz communtity, Let's say a website has multiple sub-domains with hundreds and thousands of pages. Generally we will be mentioning "primary keyword & "brand name" on every page of website. Can we do same on all pages of sub-domains to increase the authority of website for this primary keyword in Google? Or it gonna end up as negative impact if Google consider as duplicate content being mentioned same keyword and brand name on every page even on website and all pages of sub domains? Thanks

              Intermediate & Advanced SEO | | vtmoz
              0
            • dbaxa-261338

              Consolidating Multiple Domains into A Single Domain

              I have a client who's website is an amalgamation of multiple domains. jacksonhole.net is the main domain but the site passes traffic back and forth from the following domains/sites. My questions is, would it it be better for SEO to consolidate all of these domains under the single high authority domain and 301 redirect the rest or is that a really bad idea?  Thanks for your help. jacksonhole.net (Domain Authority 31) jackson-hole-rental-condos.com (Domain Authority 22) jackson-hole-rental-homes.com (Domain Authority 21) j acksonholehotelguide.com (Domain Authority 19)

              Intermediate & Advanced SEO | | dbaxa-261338
              1
            • IsaCleanse

              Redirect domain or keep separate domains in each country?

              Hi all Hoping this might be something that can be answered given the number of variables 🙂 
              My main site is www.isacleanse.com.au (Obviously targeted to Australian Market) and also www.isacleanse.co.nz targeted to NZ. The main Keywords im targeting are 'Isagenix' for both and also Isagenix Australia, Isagenix Perth, Sydney (Australian cities) and Isagenix NZ, Isagenix New Zealand, Isagenix Auckland etc.. for NZ The Australian site gets a lot more traffic and Australian market gets a lot more searches - I also have a section www.isacleanse.com.au/isagenix-new-zealand/ on the Australian site. The question is am I best off redirrecting the .co.nz domain completley to the Australian Domain to give it extra SEO Juice?

              Intermediate & Advanced SEO | | IsaCleanse
              0
            • IrvCo_Interactive

              301s being indexed

              A client website was moved about six months ago to a new domain. At the time of the move, 301 redirects were setup from the pages on the old domain to point to the same page on the new domain. New pages were setup on the old domain for a different purpose. Now almost six months later when I do a query in google on the old domain like site:example.com 80% of the pages returned are 301 redirects to the new domain. I would have expected this to go away by now. I tried removing these URLs in webmaster tools but the removal requests expire and the URLs come back. Is this something we should be concerned with?

              Intermediate & Advanced SEO | | IrvCo_Interactive
              0
            • footd

              Community inside the domain or in a separate domain

              Hi there, I work for an ecommerce company as an online marketing consultant. They make kitchenware, microware and so on. The are reviewing their overall strategy and as such they want to build up a community. Ideally, they would want to have the community in a separate domain. This domain wouldn't have the logo of the brand. This community wouldn't promote the brand itself. The brand would post content occassionally and link the store domain. The reasoning of this approach is to not interfere in the way of the community users and also the fact that the branded traffic acquired doesn't end up buying at the store I like this approach but I am concerned because the brand is not that big to have two domains separated and lose all the authority associated with one strong domain. I would definitely have everything under the same domain, store and community, otherwise we would have to acquire traffic for two domains. 1. What do you think of both scenarios, one domain versus two? Which one is better? 2. Do you know any examples of ecommerce companies with successful communities within the store domain? Thanks and regards

              Intermediate & Advanced SEO | | footd
              0
            • Bio-RadAbs

              Can you redirect specific sub domain URLs?

              ello! We host our PDFs, Images, CSS all in a sub domain. For the question, let's call this sub.cyto.com. I've noticed a particular PDF doing really well, infact it has gathered valuable external links from high authoritative sites. To top it off, it gets good visits. I've been going back and forth with our developers to move this PDF to a subfolder structure.
              For example: www.cyto.com/document/xxxx.pdf In my perspective, if I move this and set up a permanent redirect, then all the external links the PDF gathered, link juice and future visits will be attributed to the main website. Since the PDF is existing in the subdomain, I can't even track direct visits nor get the link juice. It appears in top position of Google as well. My developer says it is better to keep images, pdf, css in the subdomain. I see his point and an idea I have is to: convert the pdf to a webpage. Set up a 301 redirect from the existing subdomain to this webpage Upload the pdf with a new name and link to it from the webpage, so users can download if they choose to. This should give me the existing rank juice. However, my question is whether you can set up a 301 redirect for just a single subdomain URL to a folder structure URL? sub.cyto.com/xxx.pdf to www.cyto.com/document/xxxx.pdf?

              Intermediate & Advanced SEO | | Bio-RadAbs
              0
            • essdee

              Merging Domains... Sub-domains, Directories or Seperate Sites?

              Hello! I am hoping you can help me decide the best path to take here... A little background: I'm moving to a new company that has three old domains (the oldest is 10 years old), which get a lot of traffic from their e-letters. Until recently they have not cared about SEO. So the websites have some structural, coding, URL and other issues. The sites are indexed, but have a problem getting crawled and/or indexed for new content - haven't delved into this yet but am certain I will be able to fix any of these issues. These three domains are PR4, PR4, PR5 and contain hundreds of unique articles. Here's the question... They want to move these three sites **to their main company site (PR4) and create sub domains for each one. ** I am wondering if this is a good idea or not. I have merged sites before (creating categories and/or directories) and the end result is that the ONE big site, is much for effective than TWO smaller, less authoritative sites. But the sub domain idea is something I am unsure about from an SEO perspective. Should we do this with sub domains? Or do you think we should keep the sites separate? How do Panda and Penguin play into this? Thanks in advance for the help! SD P.S. I'm not a huge advocate in using PR as a measurement tool, but since I can't reveal the actual domains, I figured I would list it as a reference point.

              Intermediate & Advanced SEO | | essdee
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            • Digital Marketers
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.