undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • MozCon
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Digital Marketers
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    Track AI Overviews in Keyword Research
    Moz Pro

    Track AI Overviews in Keyword Research

    Try it free!
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Digital Marketers

      Simplify SEO tasks to save time and grow your traffic.

    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. Technical SEO
  4. Allow or Disallow First in Robots.txt

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Allow or Disallow First in Robots.txt

Technical SEO
7
12
30.8k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • irvingw
    irvingw last edited by May 4, 2012, 12:35 PM

    If I want to override a Disallow directive in robots.txt with an Allow command, do I have the Allow command before or after the Disallow command?

    example:

    Allow: /models/ford///page*

    Disallow: /models////page

    1 Reply Last reply Reply Quote 0
    • Net66SEO
      Net66SEO last edited by Feb 10, 2015, 7:53 AM Feb 10, 2015, 7:53 AM

      Just caught this a bit late and probably to late to add something but my two pence is test it in Webmaster Tools, via Crawl -> Robot.txt tester - if you've not used this before simply add the url you want to test and Google highlights the directive that allows or disallows it.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,about a year
      • fablau
        fablau @Cyrus-Shepard last edited by Dec 16, 2013, 6:00 PM Dec 16, 2013, 6:00 PM

        Thank you Cyrus, yes, I have tried your suggested robots.txt checker and despite it validates the file, it shows me a couple of warnings about the "unusual" use of wildcard. It is my understanding that I would probably need to discuss all this with Google folks directly.

        Thank you for you answer... and, yes Keri, I know this is a old thread, but still useful today!

        Thanks 🙂

        1 Reply Last reply Reply Quote 0
        • Cyrus-Shepard
          Cyrus-Shepard @fablau last edited by Dec 16, 2013, 4:17 PM Dec 16, 2013, 4:17 PM

          Can't say with 100% confidence, but sounds like it might work. You could always upload it to a server and use a robots.txt checker to validate, although sometimes the validator tools may incorporate slight differences in edge cases like this that make them moot.

          fablau 1 Reply Last reply Dec 16, 2013, 6:00 PM Reply Quote 1
          • KeriMorgret
            KeriMorgret @fablau last edited by Dec 16, 2013, 4:02 PM Dec 16, 2013, 4:02 PM

            Just a quick note, this question is actually from spring of 2012.

            1 Reply Last reply Reply Quote 0
            • fablau
              fablau last edited by Dec 16, 2013, 3:53 PM Dec 16, 2013, 3:53 PM

              What about something like:

              allow: /directory/$

              disallow: /directory/*

              Where I want this to be indexed:

              http://www.mysite.com/directory/

              But not this:

              http://www.mysite.com/directory/sub-directory/

              Ideas?

              KeriMorgret Cyrus-Shepard 2 Replies Last reply Dec 16, 2013, 4:17 PM Reply Quote 0
              • topic:timeago_earlier,2 years
              • irvingw
                irvingw @Cyrus-Shepard last edited by May 6, 2012, 9:18 AM May 6, 2012, 9:18 AM

                I really appreciate all that effort you put in to ensure your method was correct. many thanks.

                1 Reply Last reply Reply Quote 0
                • Cyrus-Shepard
                  Cyrus-Shepard last edited by May 6, 2012, 9:18 AM May 6, 2012, 3:05 AM

                  Interesting question - I've had this discussion a couple of times with different SEOs. Here's my best understanding: There are actually 2 different answers - one if you are talking about Google, and one for every other search engine.

                  For most search engines, the "Allow" should come first. This is because the first matching pattern always wins, for the reasons Geoff stated.

                  But Google is different. They state:

                  "At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined."

                  Robots.txt Specifications - Webmasters — Google Developers

                  So for Google, order is not important, only the specificity of the rule based on the length of the entry. But the order of precedence for rules with wildcards is undefined.

                  This last part is important, because your directives contain wildcards. If I'm reading this right, your particular directives:

                  Allow: /models/ford///page*

                  Disallow: /models////pageSo if it's "undefined" which directive will Google follow, if order isn't important? Fortunately, there's a simple way to find out.Google Webmaster allows you to test any robots.txt file. I created a dummy file based on your rules, In this case, your directives worked perfectly no matter what order I put them in.

                  | http://cyrusshepard.com/models/ford/test/test/pages | Allowed by line 2: Allow: /models/ford///page* | Allowed by line 2: Allow: /models/ford///page* |
                  | http://cyrusshepard.com/models/chevy/test/test/pages | Blocked by line 3: Disallow: /models////page | Blocked by line 3: Disallow: /models////page |

                  So, to summarize:1. Always put Allow directives first, as most search engines follow the "first rule counts" rule.2. Google doesn't care about order, but rather the specificity based on the length of the entry.3. The order of precedence for rules with wildcards is undefined.4. When in doubt, check your robots.txt file in Google Webmaster tools.Hope this helps.(sorry for the very long answer which basically says you were right all along 🙂

                  irvingw 1 Reply Last reply May 6, 2012, 9:18 AM Reply Quote 3
                  • NakulGoyal
                    NakulGoyal @irvingw last edited by May 4, 2012, 1:40 PM May 4, 2012, 1:38 PM

                    I understand your concern. I am basing my answer based on the fact that if you don't have a robots.txt at all, Google will still crawl you, which means its an allow by default. So all that matters in my opinion is the disallow, but because you need an allow from the wildcard disallow, you could allow that and disallow next.

                    Honestly, I don't think it matters. If you think the way a bot would work, it's not like robots.txt 1 line is read, then the bot goes crawling and then comes back reads the next line and so on. Does that make sense ? It reads all the lines in the robots.txt and then follows the directives. But to be sure, you can do either of the scenarios and see for yourself. I am sure the results would be same either way.

                    1 Reply Last reply Reply Quote 1
                    • zigojacko
                      zigojacko last edited by May 4, 2012, 1:40 PM May 4, 2012, 1:34 PM

                      The allow directives need to come before the disallow directives for the same directory/file paths. (I have never personally tested this although it makes logical sense to instruct a robot to access one particular path within a directory structure before it sees that it is blocked from crawling that directory).

                      For example:-

                      Allow: /profiles

                      Disallow: /s2/profiles/me

                      Allow: /s2/profiles

                      Allow: /s2/photos

                      Allow: /s2/static

                      Disallow: /s2

                      As per how Google have formatted their robots.txt.

                      1 Reply Last reply Reply Quote 2
                      • irvingw
                        irvingw @NakulGoyal last edited by May 4, 2012, 1:31 PM May 4, 2012, 1:31 PM

                        Thanks. I want to make sure I get this right in a syntax universally understood by all engines. I have seen webmasters all over the place on this one with some saying that crawlers use a first matching rule and others that say that crawlers use a last matching rule. I am almost thinking to have the allow command twice - before and after, to cover all bases.

                        NakulGoyal 1 Reply Last reply May 4, 2012, 1:38 PM Reply Quote 0
                        • NakulGoyal
                          NakulGoyal last edited by May 4, 2012, 1:21 PM May 4, 2012, 1:21 PM

                          I don't think it matters, but I think I would disallow first, because by default everything is an Allow.

                          irvingw 1 Reply Last reply May 4, 2012, 1:31 PM Reply Quote 0
                          • 1 / 1
                          1 out of 12
                          • First post
                            1/12
                            Last post

                          Got a burning SEO question?

                          Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                          Start my free trial


                          Browse Questions

                          Explore more categories

                          • Moz Tools

                            Chat with the community about the Moz tools.

                          • SEO Tactics

                            Discuss the SEO process with fellow marketers

                          • Community

                            Discuss industry events, jobs, and news!

                          • Digital Marketing

                            Chat about tactics outside of SEO

                          • Research & Trends

                            Dive into research and trends in the search industry.

                          • Support

                            Connect on product support and feature requests.

                          • See all categories

                          Related Questions

                          • Nomader

                            Crawl solutions for landing pages that don't contain a robots.txt file?

                            My site (www.nomader.com) is currently built on Instapage, which does not offer the ability to add a robots.txt file. I plan to migrate to a Shopify site in the coming months, but for now the Instapage site is my primary website. In the interim, would you suggest that I manually request a Google crawl through the search console tool? If so, how often? Any other suggestions for countering this Meta Noindex issue?

                            Technical SEO | May 2, 2019, 3:41 AM | Nomader
                            1
                          • AmandaBridge

                            Disallow wildcard match in Robots.txt

                            This is in my robots.txt file, does anyone know what this is supposed to accomplish, it doesn't appear to be blocking URLs with question marks Disallow: /?crawler=1
                            Disallow: /?mobile=1 Thank you

                            Technical SEO | Aug 28, 2018, 9:50 AM | AmandaBridge
                            0
                          • sjbridle

                            Do I need a separate robots.txt file for my shop subdomain?

                            Hello Mozzers! Apologies if this question has been asked before, but I couldn't find an answer so here goes... Currently I have one robots.txt file hosted at https://www.mysitename.org.uk/robots.txt We host our shop on a separate subdomain https://shop.mysitename.org.uk Do I need a separate robots.txt file for my subdomain? (Some Google searches are telling me yes and some no and I've become awfully confused!

                            Technical SEO | Sep 6, 2016, 11:16 AM | sjbridle
                            0
                          • Exotissimo

                            Parked domain is first in search results

                            We have several brand related domains which are parked and pointing to our main website. Some of these websites are redirecting using a 302 (don't ask, that's a whole other story), but these are being changed. But it shouldn't matter what type of redirect they are no? Since there has never been any traffic and they are not indexed? But it seems that one of them was indexed: exotravel.vn. A search for our brand name or the previous brand name (exotravel and exotissimo) brings up this parked domain first! How can that be? The domain has never been used and has no backlinks. exotravel.vn is redirecting and I submitted a change of address weeks ago to Google, but its still coming up first in all brand name searches for exotissimo or exotravel.

                            Technical SEO | Feb 17, 2015, 10:08 AM | Exotissimo
                            0
                          • imaginex

                            Should I block Map pages with robots.txt?

                            Hello, I have a website that was started in 1999. On the website I have map pages for each of the offices listed on my site, for which there are about 120. Each of the 120 maps is in a whole separate html page. There is no content in the page other than the map. I know all of the offices love having the map pages so I don't want to remove the pages. So, my question is would these pages with no real content be hurting the rankings of the other pages on our site? Therefore, should I block the pages with my robots.txt? Would I also have to remove these pages (in webmaster tools?) from Google for blocking by robots.txt to really work? I appreciate your feedback, thanks!

                            Technical SEO | Oct 29, 2014, 3:37 PM | imaginex
                            0
                          • RocketZando

                            Empty Meta Robots Directive - Harmful?

                            Hi, We had a coding update and a side-effect of that was that our directive was emptied, in other words it now reads as: on all of the site. I've since noticed that Google's cache date on all of the pages - at least, the ones I tested - have a Cached date of no later than 17 December '12 - that's the Monday after the directive was removed on mass. So, A, does anyone have solid evidence of an empty directive causing problems? Past experience, Matt Cutts, Fishkin quote, etc. And then B - It seems fairly well correlated but, does my entire site's homogenous Cached date point to this tag removal? Or is it fairly normal to have a particular cache date across a large site (we're a large ecommerce site). Our site: http://www.zando.co.za/ I'm having the directive reinstated as soon as Dev permitting. And then, for extra credit, is there a way with Google's API, or perhaps some other tool, to run an arbitrary list and retrieve Cached dates? I'd want to do this for diagnosis purposes and preferably in a way that OK with Google. I'd avoid CURLing for the cached URL and scraping out that dates with BASH, or any such kind of thing. Cheers,

                            Technical SEO | Dec 27, 2012, 8:30 AM | RocketZando
                            0
                          • MirandaP

                            How to allow googlebot past paywall

                            Does anyone know of any ways or ideas to allow Google/Bing etc. to index your content, but have it behind a paywall for users?

                            Technical SEO | May 1, 2012, 5:15 AM | MirandaP
                            0
                          • themegroup

                            Robots.txt file getting a 500 error - is this a problem?

                            Hello all! While doing some routine health checks on a few of our client sites, I spotted that a new client of ours - who's website was not designed built by us - is returning a 500 internal server error when I try to look at the robots.txt file. As we don't host / maintain their site, I would have to go through their head office to get this changed, which isn't a problem but I just wanted to check whether this error will actually be having a negative effect on their site / whether there's a benefit to getting this changed? Thanks in advance!

                            Technical SEO | Sep 6, 2011, 8:08 AM | themegroup
                            0

                          Get started with Moz Pro!

                          Unlock the power of advanced SEO tools and data-driven insights.

                          Start my free trial
                          Products
                          • Moz Pro
                          • Moz Local
                          • Moz API
                          • Moz Data
                          • STAT
                          • Product Updates
                          Moz Solutions
                          • SMB Solutions
                          • Agency Solutions
                          • Enterprise Solutions
                          • Digital Marketers
                          Free SEO Tools
                          • Domain Authority Checker
                          • Link Explorer
                          • Keyword Explorer
                          • Competitive Research
                          • Brand Authority Checker
                          • Local Citation Checker
                          • MozBar Extension
                          • MozCast
                          Resources
                          • Blog
                          • SEO Learning Center
                          • Help Hub
                          • Beginner's Guide to SEO
                          • How-to Guides
                          • Moz Academy
                          • API Docs
                          About Moz
                          • About
                          • Team
                          • Careers
                          • Contact
                          Why Moz
                          • Case Studies
                          • Testimonials
                          Get Involved
                          • Become an Affiliate
                          • MozCon
                          • Webinars
                          • Practical Marketer Series
                          • MozPod
                          Connect with us

                          Contact the Help team

                          Join our newsletter
                          Moz logo
                          © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                          • Accessibility
                          • Terms of Use
                          • Privacy

                          Looks like your connection to Moz was lost, please wait while we try to reconnect.