undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • SEO Q&A

      Insights & discussions from an SEO community of 500,000+.

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Intermediate & Advanced SEO
  4. Block an entire subdomain with robots.txt?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Block an entire subdomain with robots.txt?

Intermediate & Advanced SEO
5
16
111.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • kylesuss
    kylesuss last edited by Aug 18, 2011, 2:00 PM

    Is it possible to block an entire subdomain with robots.txt?

    I write for a blog that has their root domain as well as a subdomain pointing to the exact same IP. Getting rid of the option is not an option so I'd like to explore other options to avoid duplicate content. Any ideas?

    1 Reply Last reply Reply Quote 12
    • kylesuss
      kylesuss @kylesuss last edited by Sep 2, 2011, 12:28 PM Sep 2, 2011, 12:28 PM

      Awesome! That did the trick -- thanks for your help. The site is no longer listed 🙂

      1 Reply Last reply Reply Quote 1
      • sprynewmedia
        sprynewmedia @kylesuss last edited by Aug 28, 2011, 3:21 PM Aug 27, 2011, 3:11 PM

        Fact is, the robots file alone will never work (the link has a good explanation why - short form: all it does is stop the bots from indexing again).

        Best to request removal then wait a few days.

        1 Reply Last reply Reply Quote 3
        • kylesuss
          kylesuss @kylesuss last edited by Aug 27, 2011, 2:04 PM Aug 27, 2011, 2:04 PM

          Yeah. As of yet, the site has not been de-indexed. We placed the conditional rule in htaccess and are getting different robots.txt files for the domain and subdomain -- so that works. But I've never done this before so I don't know how long it's supposed to take?

          I'll try to verify via Webmaster Tools to speed up the process. Thanks

          1 Reply Last reply Reply Quote 0
          • sprynewmedia
            sprynewmedia @kylesuss last edited by Aug 27, 2011, 2:29 PM Aug 27, 2011, 2:00 PM

            You should do a remove request in Google Webmaster Tools.  You have to first verify the sub-domain then request the removal.

            See this post on why the robots file alone won't work...

            http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts

            1 Reply Last reply Reply Quote 1
            • kylesuss
              kylesuss @kylesuss last edited by Aug 22, 2011, 12:34 PM Aug 22, 2011, 12:33 PM

              Awesome. We used your second idea and so far it looks like it is working exactly how we want. Thanks for the idea.

              Will report back to confirm that the subdomain has been de-indexed.

              1 Reply Last reply Reply Quote 0
              • sprynewmedia
                sprynewmedia @kylesuss last edited by Aug 18, 2011, 8:31 PM Aug 18, 2011, 8:31 PM

                Option 1 could come with a small performance hit if you have a lot of txt files being used on the server.

                There shouldn't be any negative side effects to option 2 if the rewrite is clean (IE not accidently a redirect) and the content of the two files are robots compliant.

                Good luck

                1 Reply Last reply Reply Quote 2
                • kylesuss
                  kylesuss @sprynewmedia last edited by Aug 18, 2011, 8:14 PM Aug 18, 2011, 8:14 PM

                  Thanks for the suggestion. I'll definitely have to do a bit more research into this one to make sure that it doesn't have any negative side effects before implementation

                  sprynewmedia kylesuss 6 Replies Last reply Sep 2, 2011, 12:28 PM Reply Quote 0
                  • kylesuss
                    kylesuss @john4math last edited by Aug 18, 2011, 8:12 PM Aug 18, 2011, 8:12 PM

                    We have a plugin right now that places canonical tags, but unfortunately, the canonical for the subdomain points to the subdomain. I'll look around to see if I can tweak the settings

                    1 Reply Last reply Reply Quote 0
                    • sprynewmedia
                      sprynewmedia last edited by Aug 18, 2011, 4:20 PM Aug 18, 2011, 3:44 PM

                      Sounds like (from other discussions) you may be stuck requiring a dynamic robot.txt file which detects what domain the bot is on and changes the content accordingly.  This means the server has to run all .txt file as (I presume) PHP.

                      Or, you could conditionally rewrite the /robot.txt URL to a new file according to sub-domain

                      RewriteEngine on
                      RewriteCond %{HTTP_HOST} ^subdomain.website.com$
                      RewriteRule ^robotx.txt$ robots-subdomain.txt

                      Then add:

                      User-agent: *
                      Disallow: /

                      to the robots-subdomain.txt file

                      (untested)

                      kylesuss 1 Reply Last reply Aug 18, 2011, 8:14 PM Reply Quote 2
                      • john4math
                        john4math last edited by Aug 18, 2011, 3:17 PM Aug 18, 2011, 3:17 PM

                        Placing canonical tags isn't an option?  Detect that the page is being viewed through the subdomain, and if so, write the canonical tag on the page back to the root domain?

                        Or, just place a canonical tag on every page pointing back to the root domain (so the subdomain and root domain pages would both have them).  Apparently, it's ok to have a canonical tag on a page pointing to itself.  I haven't tried this, but if Matt Cutts says it's ok...

                        kylesuss 1 Reply Last reply Aug 18, 2011, 8:12 PM Reply Quote 1
                        • kylesuss
                          kylesuss @AdoptionHelp last edited by Aug 18, 2011, 3:06 PM Aug 18, 2011, 3:06 PM

                          Hey Ryan,

                          I wasn't directly involved with the decision to create the subdomain, but I'm told that it is necessary to create in order to bypass certain elements that were affecting the root domain.

                          Nevertheless, it is a blog and the users now need to login to the subdomain in order to access the Wordpress backend to bypass those elements. Traffic for the site still goes to the root domain.

                          1 Reply Last reply Reply Quote 0
                          • AdoptionHelp
                            AdoptionHelp last edited by Aug 18, 2011, 2:34 PM Aug 18, 2011, 2:34 PM

                            They both point to the same location on the server? So there's not a different folder for the subdomain?

                            If that's the case then I suggest adding a rule to your htaccess file to 301 the subdomain back to the main domain in exactly the same way people redirect from non-www to www or vice-versa. However, you should ask why the server is configured to have a duplicate subdomain? You might just edit your apache settings to get rid of that subdomain (usually done through a cpanel interface).

                            Here is what your htaccess might look like:

                            <ifmodule mod_rewrite.c="">RewriteEngine on
                              # Redirect non-www to wwww
                              RewriteCond %{HTTP_HOST} !^www.mydomain.org [NC]
                              RewriteRule ^(.*)$ http://www.mydomain.org/$1 [R=301,L]</ifmodule>

                            kylesuss 1 Reply Last reply Aug 18, 2011, 3:06 PM Reply Quote 2
                            • AndyKuiper
                              AndyKuiper last edited by Aug 18, 2011, 2:24 PM Aug 18, 2011, 2:24 PM

                              Not to me LOL 🙂 I think you'll need someone with a bit more expertise in this area than I to assist in this case. Kyle, I'm sorry I couldn't offer more assistance... but I don't want to tell you something if I'm not 100% sure. I suspect one of the many bright SEOmozer's will quickly come to the rescue on this one.

                              Andy 🙂

                              1 Reply Last reply Reply Quote 1
                              • kylesuss
                                kylesuss @AndyKuiper last edited by Aug 18, 2011, 2:15 PM Aug 18, 2011, 2:15 PM

                                Hey Andy,

                                Herein lies the problem. Since the domain and subdomain point to the exact same place, they both utilize the same robots.txt file.

                                Does that make sense?

                                1 Reply Last reply Reply Quote 0
                                • AndyKuiper
                                  AndyKuiper last edited by Aug 18, 2011, 2:14 PM Aug 18, 2011, 2:14 PM

                                  Hi Kyle 🙂 Yes, you can block an entire subdomain via robots.txt, however you'll need to create a robots.txt file and place it in the root of the subdomain, then add the code to direct the bots to stay away from the entire subdomain's content.

                                  User-agent: *
                                  Disallow: /

                                  hope this helps 🙂

                                  kylesuss 1 Reply Last reply Aug 18, 2011, 2:15 PM Reply Quote 3
                                  • 1 / 1
                                  1 out of 16
                                  • First post
                                    1/16
                                    Last post

                                  Got a burning SEO question?

                                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                                  Start my free trial


                                  Browse Questions

                                  Explore more categories

                                  • Moz Tools

                                    Chat with the community about the Moz tools.

                                  • SEO Tactics

                                    Discuss the SEO process with fellow marketers

                                  • Community

                                    Discuss industry events, jobs, and news!

                                  • Digital Marketing

                                    Chat about tactics outside of SEO

                                  • Research & Trends

                                    Dive into research and trends in the search industry.

                                  • Support

                                    Connect on product support and feature requests.

                                  • See all categories

                                  Related Questions

                                  • Kingalan1

                                    Meta Robot Tag:Index, Follow, Noodp, Noydir

                                    When should "Noodp" and "Noydir" meta robot tag be used? I have hundreds or URLs for real estate listings on my site that simply use "Index", Follow" without using Noodp and Noydir. Should the listing pages use these Noodp and Noydr also? All major landing pages use Index, Follow, Noodp, Noydir. Is this the best setting in terms of ranking and SEO. Thanks, Alan

                                    Intermediate & Advanced SEO | Nov 20, 2020, 6:27 AM | Kingalan1
                                    0
                                  • bizzer

                                    Does blocking foreign country IP traffic to site, hurt my SEO / US Google rankings?

                                    I have a website is is only of interest to US visitors.  99% (at least) of Adsense income is from the US.  But I'm getting constant attempts by hackers to login to my admin account.  I have countermeasures fo combat that and am initiating others. But here's my question: I am considering not allowing any non US, or at least any non-North American, traffic to the site via a Wordpress plugin that does this.  I know it will not affect my business negatively, directly.  However, are there any ramifications of the Google bots of these blocked countries not being able to access my site?  Does it affect the rankings of my site in the US Google searches. At the very least I could block China, Russia and some eastern European countries.

                                    Intermediate & Advanced SEO | Sep 15, 2014, 4:18 PM | bizzer
                                    0
                                  • WebServiceConsulting.com

                                    Dev Subdomain Pages Indexed - How to Remove

                                    I own a website (domain.com) and used the subdomain "dev.domain.com" while adding a new section to the site (as a development link). I forgot to block the dev.domain.com in my robots file, and google indexed all of the dev pages (around 100 of them). I blocked the site (dev.domain.com) in robots, and then proceeded to just delete the entire subdomain altogether. It's been about a week now and I still see the subdomain pages indexed on Google. How do I get these pages removed from Google? Are they causing duplicate content/title issues, or does Google know that it's a development subdomain and it's just taking time for them to recognize that I deleted it already?

                                    Intermediate & Advanced SEO | Apr 30, 2014, 10:56 AM | WebServiceConsulting.com
                                    0
                                  • bjs2010

                                    Meta NoIndex tag and Robots Disallow

                                    Hi all, I hope you can spend some time to answer my first of a few questions 🙂 We are running a Magento site - layered/faceted navigation nightmare has created thousands of duplicate URLS! Anyway, during my process to tackle the issue, I disallowed in Robots.txt anything in the querystring that was not a p (allowed this for pagination). After checking some pages in Google, I did a site:www.mydomain.com/specificpage.html and a few duplicates came up along with the original with
                                    "There is no information about this page because it is blocked by robots.txt" So I had added in Meta Noindex, follow on all these duplicates also but I guess it wasnt being read because of Robots.txt. So coming to my question. Did robots.txt block access to these pages? If so, were these already in the index and after disallowing it with robots, Googlebot could not read Meta No index? Does Meta Noindex Follow on pages actually help Googlebot decide to remove these pages from index? I thought Robots would stop and prevent indexation? But I've read this:
                                    "Noindex is a funny thing, it actually doesn’t mean “You can’t index this”, it means “You can’t show this in search results”. Robots.txt disallow means “You can’t index this” but it doesn’t mean “You can’t show it in the search results”. I'm a bit confused about how to use these in both preventing duplicate content in the first place and then helping to address dupe content once it's already in the index. Thanks! B

                                    Intermediate & Advanced SEO | May 15, 2013, 10:18 PM | bjs2010
                                    0
                                  • gregelwell

                                    Could you use a robots.txt file to disalow a duplicate content page from being crawled?

                                    A website has duplicate content pages to make it easier for users to find the information from a couple spots in the site navigation. Site owner would like to keep it this way without hurting SEO. I've thought of using the robots.txt file to disallow search engines from crawling one of the pages. Would you think this is a workable/acceptable solution?

                                    Intermediate & Advanced SEO | Jun 4, 2012, 9:15 PM | gregelwell
                                    0
                                  • Webrevolve

                                    Franchise sites on subdomains

                                    I've been asked by a client to optimise a a webpage for a location i.e. London. Turns out that the location is actually a franchise of the main company. When the company launch a new franchise, so far they have simply added a new page to the main site, for example: mysite.co.uk/sub-folder/london They have so far done this for 10 or so franchises and task someone with optimising that page for their main keyword + location. I think I know the answer to this, but would like to get a back up / additional info on it in terms of ranking / seo benefits. I am going to suggest the idea of using a subdomain for each location, example: london.mysite.co.uk Would this be the correct approach. If you think yes, why? Many thanks,

                                    Intermediate & Advanced SEO | Mar 5, 2012, 3:11 PM | Webrevolve
                                    0
                                  • Ryan-Ricketts

                                    Should we stream videos from a subdomain or subfolder?

                                    This is causing some hand-wrangling and I need some help. Let's say we were developing rich video content for our products and our agency is hosting the content on a new external server. There are already plans to link to these videos from product detail pages, category pages, and landing pages on our main website: www.example.com. Would it be better to create a new subdomain or to use a subfolder with a reverse proxy technique for this video content? It's not truly a microsite and will not have a gallery page to navigate the various videos. For example, would it be better to use this: video.example.com/ ...or this (executed with a reverse proxy😞 www.example.com/video/ Of course, regardless of whether this new content will live on a subdomain or within a subfolder, we plan on creating a video Sitemap using guidance from Justin Hammack in this terrific post.

                                    Intermediate & Advanced SEO | Feb 17, 2012, 4:16 PM | Ryan-Ricketts
                                    1
                                  • seo123456

                                    Using 2 wildcards in the robots.txt file

                                    I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string. So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it?  So something like /_Q1.  Will that pickup and block every  URL with those characters in the string? Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1.  So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on? Thanks.

                                    Intermediate & Advanced SEO | Jan 20, 2012, 5:25 AM | seo123456
                                    0

                                  Get started with Moz Pro!

                                  Unlock the power of advanced SEO tools and data-driven insights.

                                  Start my free trial
                                  Products
                                  • Moz Pro
                                  • Moz Local
                                  • Moz API
                                  • Moz Data
                                  • STAT
                                  • Product Updates
                                  Moz Solutions
                                  • SMB Solutions
                                  • Agency Solutions
                                  • Enterprise Solutions
                                  Free SEO Tools
                                  • Domain Authority Checker
                                  • Link Explorer
                                  • Keyword Explorer
                                  • Competitive Research
                                  • Brand Authority Checker
                                  • Local Citation Checker
                                  • MozBar Extension
                                  • MozCast
                                  Resources
                                  • Blog
                                  • SEO Learning Center
                                  • Help Hub
                                  • Beginner's Guide to SEO
                                  • How-to Guides
                                  • Moz Academy
                                  • API Docs
                                  About Moz
                                  • About
                                  • Team
                                  • Careers
                                  • Contact
                                  Why Moz
                                  • Case Studies
                                  • Testimonials
                                  Get Involved
                                  • Become an Affiliate
                                  • MozCon
                                  • Webinars
                                  • Practical Marketer Series
                                  • MozPod
                                  Connect with us

                                  Contact the Help team

                                  Join our newsletter

                                  Access all your tools in one place. Whether you're tracking progress or analyzing data, everything you need is at your fingertips.

                                  Moz logo
                                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                                  • Accessibility
                                  • Terms of Use
                                  • Privacy

                                  Looks like your connection to Moz was lost, please wait while we try to reconnect.