undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. SEO Tactics
  3. On-Page Optimization
  4. How do i block an entire category/directory with robots.txt?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

How do i block an entire category/directory with robots.txt?

On-Page Optimization
2
3
4.2k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • bricerhodes
    bricerhodes last edited by Mar 28, 2013, 2:16 AM

    Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.

    The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.

    I'm not really sure how i'd type it into the robots.txt file, or where to place the file.

    any help would be appreciated thanks

    1 Reply Last reply Reply Quote 0
    • bricerhodes
      bricerhodes last edited by Mar 28, 2013, 5:29 PM Mar 28, 2013, 5:29 PM

      Thanks for the detailed answer, i will give it a try!

      1 Reply Last reply Reply Quote 0
      • LynnPatchett
        LynnPatchett last edited by Mar 28, 2013, 5:18 AM Mar 28, 2013, 5:18 AM

        Hi

        This should do it, you place the robots.txt in the root directory of your site.

        User-agent: * Disallow: /product-category/

        You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt

        As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.

        Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).

        It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.

        Hope that helps!

        1 Reply Last reply Reply Quote 3
        • 1 / 1
        1 out of 3
        • First post
          1/3
          Last post

        Got a burning SEO question?

        Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


        Start my free trial


        Browse Questions

        Explore more categories

        • Moz Tools

          Chat with the community about the Moz tools.

        • SEO Tactics

          Discuss the SEO process with fellow marketers

        • Community

          Discuss industry events, jobs, and news!

        • Digital Marketing

          Chat about tactics outside of SEO

        • Research & Trends

          Dive into research and trends in the search industry.

        • Support

          Connect on product support and feature requests.

        • See all categories

        Related Questions

        • mbmozmb

          What is the best meta description for Category Pages, Tag Pages and Main Article?

          Hi, I want to index all my categories and tags. But I fear about duplicating the meta description. for example: I have a tag name "Learn Stock Market", a category name "Learning", and a main article "What is Stock Market". What is your suggestion for meta description of these three pages that looks great for seo google?

          On-Page Optimization | Jun 9, 2020, 12:59 PM | mbmozmb
          0
        • ahislop574

          Moz bar not working on https://www.fitness-china.com/gym-equipment-names-pictures-prices

          Moz bar not working on our website about gym equipment names https://www.fitness-china.com/gym-equipment-names-pictures-prices How long fix it?

          On-Page Optimization | Oct 14, 2023, 3:17 PM | ahislop574
          0
        • wearehappymedia

          Duplicate content with tagging and categories

          Hello, Moz is showing that a site has duplicate content - which appears to be because of tags and categories. It is a relatively new site, with only a few blog publications so far. This means that the same articles are displayed under a number of different tags and categories... Is this something I should worry about, or just wait until I have more content? The 'tag' and 'category' pages are not really pages I would expect or aim for anyone to find in google results anyway. Would be glad to here any advice / opinions on this Thanks!

          On-Page Optimization | Dec 17, 2015, 9:59 PM | wearehappymedia
          1
        • estebanseo

          Using keywords in my URL: Doing a redirect to /keyword

          My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.

          On-Page Optimization | Mar 30, 2015, 10:13 AM | estebanseo
          0
        • Ron_McCabe

          Any idea how Google is doing this? Is it schematic? http://techcrunch.com/2014/02/28/google-adds-full-restaurant-menus-to-its-search-results-pages/

          Google is now showing menus on select searches.  Any idea how they are getting this information? I would like to make sure my clients get visibility this way.

          On-Page Optimization | Mar 27, 2014, 10:30 PM | Ron_McCabe
          0
        • lbohen

          State Abbreviations and/or Fully Spelled Out?

          Should State names be abbreviated and/or fully spelled out in title tags, meta descriptions and body content? Does Google know that VT = Vermont?

          On-Page Optimization | Mar 17, 2013, 12:37 PM | lbohen
          0
        • BobGW

          Ecommerce On-Site SEO: Keywords in Category Descriptions

          Hello, I'm doing on-site SEO for a client's ecommerce site. Are 160 words enough for a category description? I'm using the keywords once at the top of the description, and once at the bottom of the description, with the ones at the bottom reworded so that they are the keywords with a different word order. I used to put the keywords in 3 times but it just feels like stuffing. Is twice, worded differently the second time, enough for a category description? Thanks.

          On-Page Optimization | Dec 28, 2012, 9:54 PM | BobGW
          0
        • BobGW

          Ecommerce - how many clicks from the home page should categories be

          My client has about 300 products in 20 categories with a lot of overlap. How many clicks from the home page should we keep the products? We're not doing pagination. I'd been told several years ago that all products should be 2 clicks or less from the home page. Is this true today? Thanks.

          On-Page Optimization | Dec 12, 2012, 7:49 PM | BobGW
          1

        Get started with Moz Pro!

        Unlock the power of advanced SEO tools and data-driven insights.

        Start my free trial
        Products
        • Moz Pro
        • Moz Local
        • Moz API
        • Moz Data
        • STAT
        • Product Updates
        Moz Solutions
        • SMB Solutions
        • Agency Solutions
        • Enterprise Solutions
        Free SEO Tools
        • Domain Authority Checker
        • Link Explorer
        • Keyword Explorer
        • Competitive Research
        • Brand Authority Checker
        • Local Citation Checker
        • MozBar Extension
        • MozCast
        Resources
        • Blog
        • SEO Learning Center
        • Help Hub
        • Beginner's Guide to SEO
        • How-to Guides
        • Moz Academy
        • API Docs
        About Moz
        • About
        • Team
        • Careers
        • Contact
        Why Moz
        • Case Studies
        • Testimonials
        Get Involved
        • Become an Affiliate
        • MozCon
        • Webinars
        • Practical Marketer Series
        • MozPod
        Connect with us

        Contact the Help team

        Join our newsletter
        Moz logo
        © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
        • Accessibility
        • Terms of Use
        • Privacy

        Looks like your connection to Moz was lost, please wait while we try to reconnect.