undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. On-Page Optimization
  4. How to exclude URL filter searches in robots.txt

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

How to exclude URL filter searches in robots.txt

On-Page Optimization
4
9
12.0k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • neenor
    neenor last edited by Sep 4, 2013, 12:35 PM

    When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505

    How can I exclude all of these filters in the robots.txt? I think it'll be:

    Disallow: /*?color=$

    Is that the correct syntax with the $ sign in it? Thanks!

    1 Reply Last reply Reply Quote 0
    • Everett
      Everett @swissmadesmt last edited by Jun 20, 2017, 5:00 PM Jun 20, 2017, 5:00 PM

      Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,7 days
      • swissmadesmt
        swissmadesmt last edited by Jun 13, 2017, 9:54 AM Jun 13, 2017, 9:54 AM

        Yes, but what about bing and rest of Search Engine?

        Everett 1 Reply Last reply Jun 20, 2017, 5:00 PM Reply Quote 0
        • topic:timeago_earlier,4 years
        • Everett
          Everett @neenor last edited by Sep 5, 2013, 1:59 PM Sep 5, 2013, 1:59 PM

          Adrian,

          I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.

          It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.

          And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.

          1. On the Webmaster Tools Home page, click the site you want.
          2. Under Crawl, click Blocked URLs.
          3. If it's not already selected, click the Test robots.txt tab.
          4. Copy the content of your robots.txt file, and paste it into the first box.
          5. In the URLs box, list the site to test against.
          6. In the User-agents list, select the user-agents you want (e.g. Googlebot)
          1 Reply Last reply Reply Quote 0
          • neenor
            neenor @Everett last edited by Sep 5, 2013, 1:31 PM Sep 5, 2013, 1:31 PM

            There certainly is a right answer to my question - I already posted it here earlier today:

            Disallow: /*?color=
             Disallow: /
            ?*manufacturer=

            Without the $ at the end which would otherwise denote the end of the URL.

            Everett 1 Reply Last reply Sep 5, 2013, 1:59 PM Reply Quote -1
            • Everett
              Everett last edited by Sep 5, 2013, 12:52 PM Sep 5, 2013, 12:52 PM

              Hello Adrian,

              The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.

              Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:

              (Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar

              )Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
              Duplicate Content: Block, Redirect or Canonical
              Guide to eCommerce Facets, Filters and Categories
              Rel Canonical How To and Why Not
              Moz.com Guide to Duplicate Content

              I don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...

              Good luck!

              neenor 1 Reply Last reply Sep 5, 2013, 1:31 PM Reply Quote 0
              • neenor
                neenor @stever999 last edited by Sep 5, 2013, 7:13 AM Sep 5, 2013, 7:13 AM

                It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)

                1 Reply Last reply Reply Quote 0
                • neenor
                  neenor last edited by Sep 5, 2013, 7:11 AM Sep 5, 2013, 7:11 AM

                  I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:

                  To exclude pages with color or manufacture in them you can use

                  Disallow: /*?color=
                   Disallow: /
                  ?*manufacturer=

                  A  question mark in your try should be omitted as it denotes the end of the url

                  1 Reply Last reply Reply Quote 0
                  • stever999
                    stever999 last edited by Sep 4, 2013, 5:39 PM Sep 4, 2013, 5:39 PM

                    Hi

                    I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.

                    Hope this helps.

                    Steve

                    neenor 1 Reply Last reply Sep 5, 2013, 7:13 AM Reply Quote 0
                    • 1 / 1
                    1 out of 9
                    • First post
                      1/9
                      Last post

                    Got a burning SEO question?

                    Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                    Start my free trial


                    Browse Questions

                    Explore more categories

                    • Moz Tools

                      Chat with the community about the Moz tools.

                    • SEO Tactics

                      Discuss the SEO process with fellow marketers

                    • Community

                      Discuss industry events, jobs, and news!

                    • Digital Marketing

                      Chat about tactics outside of SEO

                    • Research & Trends

                      Dive into research and trends in the search industry.

                    • Support

                      Connect on product support and feature requests.

                    • See all categories

                    Related Questions

                    • Safxmed

                      Filter By Category bad for seo?

                      on page optimization filter by category

                      Hello Everyone! I know that a single product should not have filter by color option since it will create duplicate content, and you have to use canonical tags to solve it. BUT how about sorting through products via category/brands?
                      Filter by category changes the URL of the General shop page (ex: hello.com/Shop/Category1022039 ). This page only displays the products within, no content/ descriptions etc unlike the original category page (ORIGINAL CATEGORY PAGE) Each of these category/brand already have their own individual pages (ex: hello.com/Shop/A). This is the page that will be optimized for content, FAQ, and ranking etc. Unlike in the url created when filtering through the categories. So technically I would have 2 URL for each Brand/Category. Would they compete with each other? What would you guys suggest. Please advise me on this. Thank You

                      On-Page Optimization | Mar 14, 2023, 9:56 AM | Safxmed
                      0
                    • Originaladam

                      How to get a mobile thumbnail image next to the search result?

                      Hey! I noticed the last few months that a lot of google searches now show an image next to your result when on mobile. i managed to get my listing to show an image before but I made some changes to the image and site and can’t seem to get Google to ahow the new image (or any image). I’ve added the image to schema mark up and still no luck. Wouod ahyone have any advice on how to get the image picked up, or st least help increase the odds? The site is https://tourstoniagarafalls.com and you can see it on search (for me it shows on the bottom of page 1) for the keyword “Niagara Falls tours“. Thanks again!

                      On-Page Optimization | Oct 19, 2018, 3:35 PM | Originaladam
                      0
                    • Substance-create

                      Will changing a URL negatively affect ranking?

                      Hello Mozzers, We would like to change the URL for a page on our website which ranks well for some our keyphrases/words. We are hoping the change of URL, through the addition of an additional keyword would help boost the rank of that URL further. At the moment out page gets 2 x A and 2 x B 1xF on the MOZ page rank tool using 5 keyphrase/word variations . One phrase ranks 4, one ranks 3 and the other 3 are 'not in the top 50' Our plan was to change the URL, using SHF404, and use 'Fetch' in the Google search console to re-submit the page to Google. Appreciate you can't give any guarantees how Google will behave, just wondered what your thoughts were on the wisdom of changing the URL in the first place? Thanks Ian

                      On-Page Optimization | Aug 30, 2019, 10:19 AM | Substance-create
                      0
                    • StoryScout

                      Is using hyphens in a URL to separate words good practice?

                      Hi guys, I have a client who wants to use a hyphen to separate two words in the URL to make each work stand out. Is is good or bad practice to use a hyphen in a URL and will it affect rankings? Thanks!

                      On-Page Optimization | Mar 3, 2015, 12:34 AM | StoryScout
                      0
                    • muminaydin

                      Url structure with dash or slash

                      Hi There We have a content website. We don't rank well category image related searches but we get quite good traffic for those keywords. Those keywords are mostly like "category images". We want to change our url structure and we have 2 options now. 1- domain.com/category/category-images 2-domain.com/category/images option 1 repeats the category name so it looks spammy option 2 doesn't really have the keyword. any ideas which one tho choose? Thanks! ps: we don't want to use domain.com/category-images (too many root link)

                      On-Page Optimization | Feb 10, 2014, 3:54 PM | muminaydin
                      0
                    • jampaper

                      How do I PERMANENTLY change an incorrect URL link with one of my keywords?

                      Hello, I received an “F” grade on my on-page report for one of my keywords. I noticed that the URL linked with that keyword was wrong. After changing it to the correct URL, I received an “A.” However, the change was not permanent. How do I permanently change the URL? I don’t see any “Save Changes” or “Apply Changes” button anywhere. Thank you!

                      On-Page Optimization | Dec 4, 2012, 6:29 PM | jampaper
                      0
                    • matt-14567

                      Does it matter if your URL ends in .net or .com?

                      Someone told me that having a URL that ends in .net (instead of .com) will hurt my site's SEO. Is that true?

                      On-Page Optimization | May 19, 2011, 2:55 PM | matt-14567
                      0
                    • DisMedia

                      How do you block development servers with robots.txt?

                      When we create client websites the urls are client.oursite.com.  Google is indexing theses sites and attaching to our domain.  How can we stop it with robots.txt?  I've heard you need to have the robots file on both the main site and the dev sites... A code sample would be groovy.  Thanks, TR

                      On-Page Optimization | May 11, 2012, 5:56 AM | DisMedia
                      0

                    Get started with Moz Pro!

                    Unlock the power of advanced SEO tools and data-driven insights.

                    Start my free trial
                    Products
                    • Moz Pro
                    • Moz Local
                    • Moz API
                    • Moz Data
                    • STAT
                    • Product Updates
                    Moz Solutions
                    • SMB Solutions
                    • Agency Solutions
                    • Enterprise Solutions
                    Free SEO Tools
                    • Domain Authority Checker
                    • Link Explorer
                    • Keyword Explorer
                    • Competitive Research
                    • Brand Authority Checker
                    • Local Citation Checker
                    • MozBar Extension
                    • MozCast
                    Resources
                    • Blog
                    • SEO Learning Center
                    • Help Hub
                    • Beginner's Guide to SEO
                    • How-to Guides
                    • Moz Academy
                    • API Docs
                    About Moz
                    • About
                    • Team
                    • Careers
                    • Contact
                    Why Moz
                    • Case Studies
                    • Testimonials
                    Get Involved
                    • Become an Affiliate
                    • MozCon
                    • Webinars
                    • Practical Marketer Series
                    • MozPod
                    Connect with us

                    Contact the Help team

                    Join our newsletter
                    Moz logo
                    © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                    • Accessibility
                    • Terms of Use
                    • Privacy

                    Looks like your connection to Moz was lost, please wait while we try to reconnect.