Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Track AI Overviews in Keyword Research
      Moz Pro

      Track AI Overviews in Keyword Research

      Try it free!
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Unlock flexible pricing & new endpoints
      Moz API

      Unlock flexible pricing & new endpoints

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Technical SEO
    4. What is the best method to block a sub-domain, e.g. staging.domain.com/ from getting indexed?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    What is the best method to block a sub-domain, e.g. staging.domain.com/ from getting indexed?

    Technical SEO
    4
    13
    7905
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • fthead9
      fthead9 last edited by

      Now that Google considers subdomains as part of the TLD I'm a little leery of testing robots.txt with something like:

      staging.domain.com
      User-agent: *
      Disallow: /

      in fear it might get the www.domain.com blocked as well. Has anyone had any success using robots.txt to block sub-domains? I know I could add a meta robots tag to the staging.domain.com pages but that would require a lot more work.

      1 Reply Last reply Reply Quote 0
      • KeriMorgret
        KeriMorgret @RyanKent last edited by

        Just make sure that when/if you copy over the staging site to the live domain that you don't copy over the robots.txt, htaccess, or whatever means you use to block that site from being indexed and thus have your shiny new site be blocked. 🙂

        1 Reply Last reply Reply Quote 1
        • RyanKent
          RyanKent @RyanKent last edited by

          I agree. The name of your subdomain being "staging" didn't register at all with me until Matt brought it up. I was offering a generic response to the subdomain question whereas I believe Matt focused on how to handle a staging site. Interesting viewpoint.

          1 Reply Last reply Reply Quote 0
          • fthead9
            fthead9 @RyanKent last edited by

            Matt/Ryan-

            Great discussion, thanks for the input. The staging.domain.com is just one of the domains we don't want indexed. Some of them still need to be accessed by the public, some like staging could be restricted to specific IPs.

            I realize after your discussion I probably should have used a different example of a sub-domain. On the other hand it might not have sparked the discussion so maybe it was a good example 😉

            1 Reply Last reply Reply Quote 0
            • bloggidy
              bloggidy @RyanKent last edited by

              .htaccess files can be placed at any directory level of a site so you can do it for just the subdomain or even just a directory of a domain.

              1 Reply Last reply Reply Quote 1
              • bloggidy
                bloggidy @fthead9 last edited by

                Staging URL's are typically only used for testing so rather than do a deny I would recommend using a specific ALLOW for only the IP addresses that should be allowed access.

                I would imagine you don't want it indexed because you don't want the rest of the world knowing about it.

                You can also use HTACCESS to use username/passwords. It is simple but you can give that to clients if that is a concern/need.

                1 Reply Last reply Reply Quote 0
                • RyanKent
                  RyanKent @fthead9 last edited by

                  Correct.

                  1 Reply Last reply Reply Quote 0
                  • RyanKent
                    RyanKent @fthead9 last edited by

                    Toren, I would not recommend that solution. There is nothing to prevent Googlebot from crawling your site via almost any IP. If you found 100 IPs used by the crawler and blocked them all, there is nothing to stop the crawler from using IP #101 next month. Once the subdomain's content is located and indexed, it will be a headache fixing the issue.

                    The best solution is always going to be a noindex meta tag on the pages you do not wish to be indexed. If that method is too much work or otherwise undesirable, you can use the robots.txt solution. There is no circumstance I can imagine where you would modify your htaccess file to block googlebot.

                    1 Reply Last reply Reply Quote 1
                    • RyanKent
                      RyanKent @bloggidy last edited by

                      Hi Matt.

                      Perhaps I misunderstood the question but I believe Toren only wishes to prevent the subdomain from being indexed. If you restrict subdomain access by IP it would prevent visitors from accessing the content which I don't believe is the goal.

                      bloggidy fthead9 RyanKent KeriMorgret 4 Replies Last reply Reply Quote 1
                      • fthead9
                        fthead9 @bloggidy last edited by

                        Interesting, hadn't thought of using htaccess to block Googlebot.Thanks for the suggestion.

                        RyanKent bloggidy 2 Replies Last reply Reply Quote 0
                        • fthead9
                          fthead9 @RyanKent last edited by

                          Thanks Ryan. So you don't see any issues with de-indexing the main site if I created a second robots.txt file, e.g.

                          http://staging.domin.com/robots.txt

                          User-agent: *
                          Disallow: /

                          That was my initial thought but when Google announced they consider sub-domains part of the TLD I was afraid it might affect the htp://www.domain.com versions of the pages. So you're saying the subdomain is basically treated like a folder you block on the primary domain?

                          RyanKent 1 Reply Last reply Reply Quote 0
                          • bloggidy
                            bloggidy last edited by

                            Use an .htaccess file to only allow from certain ip addresses or ranges.

                            Here is an article describing how: http://www.kirupa.com/html5/htaccess_tricks.htm

                            fthead9 RyanKent 2 Replies Last reply Reply Quote 1
                            • RyanKent
                              RyanKent last edited by

                              What is the best method to block a sub-domain, e.g. staging.domain.com/ from getting indexed?

                              Place a robots.txt file in the root of the subdomain.

                              User-agent: *
                              Disallow: /

                              This method will block the subdomain while leaving your primary domain unaffected.

                              fthead9 1 Reply Last reply Reply Quote 2
                              • 1 / 1
                              • First post
                                Last post

                              Got a burning SEO question?

                              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                              Start my free trial


                              Browse Questions

                              Explore more categories

                              • Moz Tools

                                Chat with the community about the Moz tools.

                              • SEO Tactics

                                Discuss the SEO process with fellow marketers

                              • Community

                                Discuss industry events, jobs, and news!

                              • Digital Marketing

                                Chat about tactics outside of SEO

                              • Research & Trends

                                Dive into research and trends in the search industry.

                              • Support

                                Connect on product support and feature requests.

                              • See all categories

                              Related Questions

                              • Siphoplait

                                Multiple CMS on one website / domain & SEO

                                For a client we would like to work with a content hub, but their website is build on a custom CMS so we are limited in our options and if we aks their web developers they ask crazy prices to help us.  So now we have the idea to build the content hub with wordpress and implement it next to their current CMS. for example on www.website.com/contenthub/ . As far as i know this is technically possible and there are no negative effects regarding SEO as long as we link the two sitemaps together. Am i right or am i missing something here?

                                Technical SEO | | Siphoplait
                                0
                              • 21centuryweb

                                Why images are not getting indexed and showing in Google webmaster

                                Hi, I would like to ask why our website images not indexing in Google. I have shared the following screenshot of the search console.  https://www.screencast.com/t/yKoCBT6Q8Upw Last week (Friday 14 Sept 2018) it was showing 23.5K out 31K were submitted and indexed by Google. But now, it is showing only 1K 😞 Can you please let me know why might this happen, why images are not getting indexed and showing in Google webmaster.

                                Technical SEO | | 21centuryweb
                                0
                              • RichHamilton_qcs

                                Best practices for types of pages not to index

                                Trying to better understand best practices for when and when not use a content="noindex".  Are there certain types of pages that we shouldn't want Google to index?  Contact form pages, privacy policy pages, internal search pages, archive pages (using wordpress).  Any thoughts would be appreciated.

                                Technical SEO | | RichHamilton_qcs
                                0
                              • advmedialab

                                Help: buy domain from Tradenames.com?

                                Hello to all, I'm Silvia. I am writing to ask if any of you know this site: tradenames.com. It is a domains broker. They contacted my client and would like to sell the .com business domain (my client currently has the .it). Does anyone know them? Thanks you for your help.

                                Technical SEO | | advmedialab
                                0
                              • Nanook1

                                Is it better to use XXX.com or XXX.com/index.html as canonical page

                                Is it better to use 301 redirects or canonical page? I suspect canonical is easier. The question is, which is the best canonical page, YYY.com or YYY.com/indexhtml? I assume YYY.com, since there will be many other pages such as YYY.com/info.html, YYY.com/services.html, etc.

                                Technical SEO | | Nanook1
                                0
                              • M_D_Golden_Peak

                                How to Remove /feed URLs from Google's Index

                                Hey everyone, I have an issue with RSS /feed URLs being indexed by Google for some of our Wordpress sites. Have a look at this Google query, and click to show omitted search results. You'll see we have 500+ /feed URLs indexed by Google, for our many category pages/etc. Here is one of the example URLs: http://www.howdesign.com/design-creativity/fonts-typography/letterforms/attachment/gilhelveticatrade/feed/. Based on this content/code of the XML page, it looks like Wordpress is generating these: <generator>http://wordpress.org/?v=3.5.2</generator> Any idea how to get them out of Google's index without 301 redirecting them? We need the Wordpress-generated RSS feeds to work for various uses. My first two thoughts are trying to work with our Development team to see if we can get a "noindex" meta robots tag on the pages, by they are dynamically-generated pages...so I'm not sure if that will be possible. Or, perhaps we can add a "feed" paramater to GWT "URL Parameters" section...but I don't want to limit Google from crawling these again...I figure I need Google to crawl them and see some code that says to get the pages out of their index...and THEN not crawl the pages anymore. I don't think the "Remove URL" feature in GWT will work, since that tool only removes URLs from the search results, not the actual Google index. FWIW, this site is using the Yoast plugin. We set every page type to "noindex" except for the homepage, Posts, Pages and Categories. We have other sites on Yoast that do not have any /feed URLs indexed by Google at all. Side note, the /robots.txt file was previously blocking crawling of the /feed URLs on this site, which is why you'll see that note in the Google SERPs when you click on the query link given in the first paragraph.

                                Technical SEO | | M_D_Golden_Peak
                                0
                              • MattAaron

                                Do Backlinks to a PDF help with overall authority/link juice for the rest of the domain?

                                We are working on a website that has some high-quality industry articles available on their website. For each article, there is an abstract with a link to the PDF which is hosted on the domain. We have found in Analytics that a lot of sites link directly to the PDF and not the webpage that has the abstract of the article. Can we get any benefit from a direct PDF link? Or do we need to modify our strategy?

                                Technical SEO | | MattAaron
                                0
                              • sboelter

                                Multiple Domains, Same IP address, redirecting to preferred domain (301) -site is still indexed under wrong domains

                                Due to acquisitions over time and the merging of many microsites into one major site, we currently have 20+ TLD's  pointing to the same IP address as our "preferred domain:"  for our consolidated website http://goo.gl/gH33w. They are all set up as 301 redirects on apache - including both the www and non www versions. When we launched this consolidated website, (April 2010) we accidentally left the settings of our site open to accept any of our domains on the same IP.  This was later fixed but unfortunately Google indexed our site under multiple of these URL's (ignoring the redirects) using the same content from our main website but swapping out the domain.  We added some additional redirects on apache to redirect these individual pages pages indexed under the wrong domain to the same page under our main domain http://goo.gl/gH33w.  This seemed to help resolve the issue and moved hundreds of pages off the index.  However, in December of 2010 we made significant changes in our external dns for our ip addresses and now since December, we see pages indexed under these redirecting domains on the rise again. If you do a search query of : site:laboratoryid.com you will see a few hundred examples of pages indexed under the wrong domain.  When you click on the link, it does redirect to the same page but under the preferred domain.  So the redirect is working and has been confirmed as 301.  But for some reason Google continues to crawl our site and index under this incorrect domains.  Why is this? Is there a setting we are missing?   These domain level and page level redirects should be decreasing the pages being indexed under the wrong domain but it appears it is doing the reverse. All of these old domains currently point to our production IP address where are preferred domain is also pointing.  Could this be the issue? None of the pages indexed today are from the old version of these sites. They only seem to be the new content from the new site but not under the preferred domain. Any insight would be much appreciated because we have tried many things without success to get this resolved.

                                Technical SEO | | sboelter
                                0

                              Get started with Moz Pro!

                              Unlock the power of advanced SEO tools and data-driven insights.

                              Start my free trial
                              Products
                              • Moz Pro
                              • Moz Local
                              • Moz API
                              • Moz Data
                              • STAT
                              • Product Updates
                              Moz Solutions
                              • SMB Solutions
                              • Agency Solutions
                              • Enterprise Solutions
                              • Digital Marketers
                              Free SEO Tools
                              • Domain Authority Checker
                              • Link Explorer
                              • Keyword Explorer
                              • Competitive Research
                              • Brand Authority Checker
                              • Local Citation Checker
                              • MozBar Extension
                              • MozCast
                              Resources
                              • Blog
                              • SEO Learning Center
                              • Help Hub
                              • Beginner's Guide to SEO
                              • How-to Guides
                              • Moz Academy
                              • API Docs
                              About Moz
                              • About
                              • Team
                              • Careers
                              • Contact
                              Why Moz
                              • Case Studies
                              • Testimonials
                              Get Involved
                              • Become an Affiliate
                              • MozCon
                              • Webinars
                              • Practical Marketer Series
                              • MozPod
                              Connect with us

                              Contact the Help team

                              Join our newsletter
                              Moz logo
                              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                              • Accessibility
                              • Terms of Use
                              • Privacy

                              Looks like your connection to Moz was lost, please wait while we try to reconnect.