Skip to content
    Moz logo Menu open Menu close
    • Products
      • Moz Pro
      • Moz Pro Home
      • Moz Local
      • Moz Local Home
      • STAT
      • Moz API
      • Moz API Home
      • Compare SEO Products
      • Moz Data
    • Free SEO Tools
      • Domain Analysis
      • Keyword Explorer
      • Link Explorer
      • Competitive Research
      • MozBar
      • More Free SEO Tools
    • Learn SEO
      • Beginner's Guide to SEO
      • SEO Learning Center
      • Moz Academy
      • MozCon
      • Webinars, Whitepapers, & Guides
    • Blog
    • Why Moz
      • Digital Marketers
      • Agency Solutions
      • Enterprise Solutions
      • Small Business Solutions
      • The Moz Story
      • New Releases
    • Log in
    • Log out
    • Products
      • Moz Pro

        Your all-in-one suite of SEO essentials.

      • Moz Local

        Raise your local SEO visibility with complete local SEO management.

      • STAT

        SERP tracking and analytics for enterprise SEO experts.

      • Moz API

        Power your SEO with our index of over 44 trillion links.

      • Compare SEO Products

        See which Moz SEO solution best meets your business needs.

      • Moz Data

        Power your SEO strategy & AI models with custom data solutions.

      Enhance Keyword Discovery with Bulk Analysis
      Moz Pro

      Enhance Keyword Discovery with Bulk Analysis

      Learn more
    • Free SEO Tools
      • Domain Analysis

        Get top competitive SEO metrics like DA, top pages and more.

      • Keyword Explorer

        Find traffic-driving keywords with our 1.25 billion+ keyword index.

      • Link Explorer

        Explore over 40 trillion links for powerful backlink data.

      • Competitive Research

        Uncover valuable insights on your organic search competitors.

      • MozBar

        See top SEO metrics for free as you browse the web.

      • More Free SEO Tools

        Explore all the free SEO tools Moz has to offer.

      NEW Keyword Suggestions by Topic
      Moz Pro

      NEW Keyword Suggestions by Topic

      Learn more
    • Learn SEO
      • Beginner's Guide to SEO

        The #1 most popular introduction to SEO, trusted by millions.

      • SEO Learning Center

        Broaden your knowledge with SEO resources for all skill levels.

      • On-Demand Webinars

        Learn modern SEO best practices from industry experts.

      • How-To Guides

        Step-by-step guides to search success from the authority on SEO.

      • Moz Academy

        Upskill and get certified with on-demand courses & certifications.

      • MozCon

        Save on Early Bird tickets and join us in London or New York City

      Access 20 years of data with flexible pricing
      Moz API

      Access 20 years of data with flexible pricing

      Find your plan
    • Blog
    • Why Moz
      • Digital Marketers

        Simplify SEO tasks to save time and grow your traffic.

      • Small Business Solutions

        Uncover insights to make smarter marketing decisions in less time.

      • Agency Solutions

        Earn & keep valuable clients with unparalleled data & insights.

      • Enterprise Solutions

        Gain a competitive edge in the ever-changing world of search.

      • The Moz Story

        Moz was the first & remains the most trusted SEO company.

      • New Releases

        Get the scoop on the latest and greatest from Moz.

      Surface actionable competitive intel
      New Feature

      Surface actionable competitive intel

      Learn More
    • Log in
      • Moz Pro
      • Moz Local
      • Moz Local Dashboard
      • Moz API
      • Moz API Dashboard
      • Moz Academy
    • Avatar
      • Moz Home
      • Notifications
      • Account & Billing
      • Manage Users
      • Community Profile
      • My Q&A
      • My Videos
      • Log Out

    The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. Home
    2. SEO Tactics
    3. Intermediate & Advanced SEO
    4. Robots.txt: how to exclude sub-directories correctly?

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Robots.txt: how to exclude sub-directories correctly?

    Intermediate & Advanced SEO
    3
    10
    53217
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with question management privileges can see it.
    • fablau
      fablau last edited by

      Hello here,

      I am trying to figure out the correct way to tell SEs to crawls this:

      http://www.mysite.com/directory/

      But not this:

      http://www.mysite.com/directory/sub-directory/

      or this:

      http://www.mysite.com/directory/sub-directory2/sub-directory/...

      But with the fact I have thousands of sub-directories with almost infinite combinations, I can't put the following definitions in a manageable way:

      disallow: /directory/sub-directory/

      disallow: /directory/sub-directory2/

      disallow: /directory/sub-directory/sub-directory/

      disallow: /directory/sub-directory2/subdirectory/

      etc...

      I would end up having thousands of definitions to disallow all the possible sub-directory combinations.

      So, is the following way a correct, better and shorter way to define what I want above:

      allow: /directory/$

      disallow: /directory/*

      Would the above work?

      Any thoughts are very welcome! Thank you in advance.

      Best,

      Fab.

      1 Reply Last reply Reply Quote 1
      • MickEdwards
        MickEdwards @sjunaidali last edited by

        I mentioned both.  You add a meta robots to noindex and remove from the sitemap.

        1 Reply Last reply Reply Quote 0
        • sjunaidali
          sjunaidali @MickEdwards last edited by

          But google is still free to index a link/page even if it is not included in xml sitemap.

          MickEdwards 1 Reply Last reply Reply Quote 0
          • MickEdwards
            MickEdwards @sjunaidali last edited by

            Install Yoast Wordpress SEO plugin and use that to restrict what is indexed and what is allowed in a sitemap.

            sjunaidali 1 Reply Last reply Reply Quote 1
            • sjunaidali
              sjunaidali @MickEdwards last edited by

              I am using wordpress, Enfold theme (themeforest).

              I want some files to be accessed by google, but those should not be indexed.

              Here is an example: http://prntscr.com/h8918o

              I have currently blocked some JS directories/files using robots.txt (check screenshot)

              But due to this I am not able to pass Mobile Friendly Test on Google: http://prntscr.com/h8925z (check screenshot)

              Is its possible to allow access, but use a tag like noindex in the robots.txt file. Or is there any other way out.

              MickEdwards 1 Reply Last reply Reply Quote 0
              • fablau
                fablau last edited by

                Yes, everything looks good, Webmaster Tools gave me the expected results with the following directives:

                allow: /directory/$

                disallow: /directory/*

                Which allows this URL:

                http://www.mysite.com/directory/

                But doesn't allow the following one:

                http://www.mysite.com/directory/sub-directory2/...

                This page also gives an update similar to mine:

                https://support.google.com/webmasters/answer/156449?hl=en

                I think I am good! Thanks 🙂

                1 Reply Last reply Reply Quote 2
                • fablau
                  fablau last edited by

                  Thank you Michael, it is my understanding then that my idea of doing this:

                  allow: /directory/$

                  disallow: /directory/*

                  Should work just fine. I will test it within Google Webmaster Tools, and let you know if any problems arise.

                  In the meantime if anyone else has more ideas about all this and can confirm me that would be great!

                  Thank you again.

                  1 Reply Last reply Reply Quote 1
                  • MickEdwards
                    MickEdwards @fablau last edited by

                    I've always stuck to Disallow and followed -

                    "This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory:"

                    http://www.robotstxt.org/robotstxt.html

                    From https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt this seems contradictory

                    | /* | equivalent to / | equivalent to / | Equivalent to "/" -- the trailing wildcard is ignored. |

                    I think this post will be very useful  for you - http://moz.com/community/q/allow-or-disallow-first-in-robots-txt

                    1 Reply Last reply Reply Quote 1
                    • fablau
                      fablau @MickEdwards last edited by

                      Thank you Michael,

                      Google and other SEs actually recognize the "allow:" command:

                      https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

                      The fact is: if I don't specify that, how can I be sure that the following single command:

                      disallow: /directory/*

                      Doesn't prevent SEs to spider the /directory/ index as I'd like to?

                      MickEdwards 1 Reply Last reply Reply Quote 0
                      • MickEdwards
                        MickEdwards last edited by

                        As long as you dont have directories somewhere in /* that you want indexed then I think that will work.  There is no allow so you don't need the first line just

                        disallow: /directory/*

                        You can test out here- https://support.google.com/webmasters/answer/156449?rd=1

                        fablau sjunaidali 2 Replies Last reply Reply Quote 0
                        • 1 / 1
                        • First post
                          Last post

                        Got a burning SEO question?

                        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                        Start my free trial


                        Browse Questions

                        Explore more categories

                        • Moz Tools

                          Chat with the community about the Moz tools.

                        • SEO Tactics

                          Discuss the SEO process with fellow marketers

                        • Community

                          Discuss industry events, jobs, and news!

                        • Digital Marketing

                          Chat about tactics outside of SEO

                        • Research & Trends

                          Dive into research and trends in the search industry.

                        • Support

                          Connect on product support and feature requests.

                        • See all categories

                        Related Questions

                        • Mat_C

                          Robots.txt blocked internal resources Wordpress

                          Hi all, We've recently migrated a Wordpress website from staging to live, but the robots.txt was deleted.  I've created the following new one: User-agent: *
                          Allow: /
                          Disallow: /wp-admin/
                          Disallow: /wp-includes/
                          Disallow: /wp-content/plugins/
                          Disallow: /wp-content/cache/
                          Disallow: /wp-content/themes/
                          Allow: /wp-admin/admin-ajax.php However, in the site audit on SemRush,  I now get the mention that a lot of pages have issues with blocked internal resources in robots.txt file. These blocked internal resources are all cached and minified css elements: links, images and scripts. Does this mean that Google won't crawl some parts of these pages with blocked resources correctly and thus won't be able to follow these links and index the images? In other words, is this any cause for concern regarding SEO? Of course I can change the robots.txt again, but will urls like https://example.com/wp-content/cache/minify/df983.js end up in the index? Thanks for your thoughts!

                          Intermediate & Advanced SEO | | Mat_C
                          2
                        • workathomecareers

                          Should comments and feeds be disallowed in robots.txt?

                          Hi My robots file is currently set up as listed below. From an SEO point of view is it good to disallow feeds, rss and comments? I feel allowing comments would be a good thing because it's new content that may rank in the search engines as the comments left on my blog often refer to questions or companies folks are searching for more information on. And the comments are added regularly. What's your take? I'm also concerned about the /page being blocked. Not sure how that benefits my blog from an SEO point of view as well. Look forward to your feedback. Thanks. Eddy User-agent: Googlebot Crawl-delay: 10 Allow: /* User-agent: * Crawl-delay: 10 Disallow: /wp- Disallow: /feed/ Disallow: /trackback/ Disallow: /rss/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /date/ Disallow: /comments/ # Allow Everything Allow: /*

                          Intermediate & Advanced SEO | | workathomecareers
                          0
                        • Bio-RadAbs

                          Can you redirect specific sub domain URLs?

                          ello! We host our PDFs, Images, CSS all in a sub domain. For the question, let's call this sub.cyto.com. I've noticed a particular PDF doing really well, infact it has gathered valuable external links from high authoritative sites. To top it off, it gets good visits. I've been going back and forth with our developers to move this PDF to a subfolder structure.
                          For example: www.cyto.com/document/xxxx.pdf In my perspective, if I move this and set up a permanent redirect, then all the external links the PDF gathered, link juice and future visits will be attributed to the main website. Since the PDF is existing in the subdomain, I can't even track direct visits nor get the link juice. It appears in top position of Google as well. My developer says it is better to keep images, pdf, css in the subdomain. I see his point and an idea I have is to: convert the pdf to a webpage. Set up a 301 redirect from the existing subdomain to this webpage Upload the pdf with a new name and link to it from the webpage, so users can download if they choose to. This should give me the existing rank juice. However, my question is whether you can set up a 301 redirect for just a single subdomain URL to a folder structure URL? sub.cyto.com/xxx.pdf to www.cyto.com/document/xxxx.pdf?

                          Intermediate & Advanced SEO | | Bio-RadAbs
                          0
                        • cos2030

                          Archiving a festival website - subdomain or directory?

                          Hi guys I look after a festival website whose program changes year in and year out. There are a handful of mainstay events in the festival which remain each year, but there are a bunch of other events which change each year around the mainstay programming.This often results in us redoing the website each year (a frustrating experience indeed!) We don't archive our past festivals online, but I'd like to start doing so for a number of reasons 1. These past festivals have historical value - they happened, and they contribute to telling the story of the festival over the years. They can also be used as useful windows into the upcoming festival. 2. The old events (while no longer running) often get many social shares, high quality links and in some instances still drive traffic. We try out best to 301 redirect these high value pages to the new festival website, but it's not always possible to find a similar alternative (so these redirects often go to the homepage) Anyway, I've noticed some festivals archive their content into a subdirectory - i.e. www.event.com/2012 However, I'm thinking it would actually be easier for my team to archive via a subdomain like 2012.event.com - and always use the www.event.com URL for the current year's event. I'm thinking universally redirecting the content would be easier, as would cloning the site / database etc. My question is - is one approach (i.e. directory vs. subdomain) better than the other? Do I need to be mindful of using a subdomain for archival purposes? Hope this all makes sense. Many thanks!

                          Intermediate & Advanced SEO | | cos2030
                          0
                        • WebRiverGroup

                          Should canonical links be included or excluded in a sitemap?

                          Our company is in the process of updating our sitemap. Should we include or exclude canonical links.

                          Intermediate & Advanced SEO | | WebRiverGroup
                          0
                        • knowyourbank

                          URL Structure for Directory Site

                          We have a directory that we're building and we're not sure if we should try to make each page an extension of the root domain or utilize sub-directories as users narrow down their selection. What is the best practice here for maximizing your SERP authority? Choice #1 - Hyphenated Architecture (no sub-folders): State Page /state/ City Page /city-state/ Business Page /business-city-state/
                          4) Location Page  /locationname-city-state/ or.... Choice #2 - Using sub-folders on drill down: State Page /state/ City Page /state/city Business Page /state/city/business/
                          4) Location Page  /locationname-city-state/ Again, just to clarify, I need help in determining what the best methodology is for achieving the greatest SEO benefits. Just by looking it would seem that choice #1 would work better because the URL's are very clear and SEF. But, at the same time it may be less intuitive for search. I'm not sure. What do you think?

                          Intermediate & Advanced SEO | | knowyourbank
                          0
                        • seo123456

                          Using 2 wildcards in the robots.txt file

                          I have a URL string which I don't want to be indexed. it includes the characters _Q1 ni the middle of the string. So in the robots.txt can I use 2 wildcards in the string to take out all of the URLs with that in it?  So something like /_Q1.  Will that pickup and block every  URL with those characters in the string? Also, this is not directly of the root, but in a secondary directory, so .com/.../_Q1.  So do I have to format the robots.txt as //_Q1* as it will be in the second folder or just using /_Q1 will pickup everything no matter what folder it is on? Thanks.

                          Intermediate & Advanced SEO | | seo123456
                          0
                        • kylesuss

                          Block an entire subdomain with robots.txt?

                          Is it possible to block an entire subdomain with robots.txt? I write for a blog that has their root domain as well as a subdomain pointing to the exact same IP. Getting rid of the option is not an option so I'd like to explore other options to avoid duplicate content. Any ideas?

                          Intermediate & Advanced SEO | | kylesuss
                          12

                        Get started with Moz Pro!

                        Unlock the power of advanced SEO tools and data-driven insights.

                        Start my free trial
                        Products
                        • Moz Pro
                        • Moz Local
                        • Moz API
                        • Moz Data
                        • STAT
                        • Product Updates
                        Moz Solutions
                        • SMB Solutions
                        • Agency Solutions
                        • Enterprise Solutions
                        • Digital Marketers
                        Free SEO Tools
                        • Domain Authority Checker
                        • Link Explorer
                        • Keyword Explorer
                        • Competitive Research
                        • Brand Authority Checker
                        • Local Citation Checker
                        • MozBar Extension
                        • MozCast
                        Resources
                        • Blog
                        • SEO Learning Center
                        • Help Hub
                        • Beginner's Guide to SEO
                        • How-to Guides
                        • Moz Academy
                        • API Docs
                        About Moz
                        • About
                        • Team
                        • Careers
                        • Contact
                        Why Moz
                        • Case Studies
                        • Testimonials
                        Get Involved
                        • Become an Affiliate
                        • MozCon
                        • Webinars
                        • Practical Marketer Series
                        • MozPod
                        Connect with us

                        Contact the Help team

                        Join our newsletter
                        Moz logo
                        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                        • Accessibility
                        • Terms of Use
                        • Privacy

                        Looks like your connection to Moz was lost, please wait while we try to reconnect.