Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I replace categories with a static page
Hello there. I want to replace all of WordPress categories with static pages so that users see a well designed and constructed presentation of all the articles within each topic instead of just a long list of excerpts. I've already done this with 2 categories and although it is hard work I can't help feeling it is a much better thing for my users. However, I'm concerned that I am embarking on this project without being totally sure that it makes sense from an Seo point of view, or whether there are any downsides I haven't thought of? My idea is that the WordPress categories are set to noindex and nofollow. Search engines should find all of my static category pages and all of the content within each category will be spidered from there instead. Just to be sure you know what I mean here is a link to a normal category - https://www.whitegoodshelp.co.uk/category/consumer/ and here is my static page replacement for it - https://www.whitegoodshelp.co.uk/consumer-rights-appliances/ Both pages contain links to all articles within the category except the one generated by WordPress is just a long paginated list, and my replacement is a proper category page, which is hopefully far more useful . Can someone please confirm that there are no downsides to this strategy? 🙂
On-Page Optimization | | Snowdune1 -
Moz bar not working on https://www.fitness-china.com/gym-equipment-names-pictures-prices
Moz bar not working on our website about gym equipment names https://www.fitness-china.com/gym-equipment-names-pictures-prices How long fix it?
On-Page Optimization | | ahislop5740 -
URL Structure on Category Pages
Hi, Currently, we having the following URL Structure o our product pages: All Products Pages: www.viatrading.com/wholesale/283/All_Products.html Category Page: www.viatrading.com/wholesale/4/Clothing.html Product Page: www.viatrading.com/wholesale/product/LOAD-HE-WOM/Assorted-High-End-Women-Clothing-Lots.html?cid=4 Since we are going to use another frontend system, we are thinking about re-working on this URL Structure, using something like this: All Products Pages: www.viatrading.com/wholesale-products/ Category Page: www.viatrading.com/wholesale-products/category/ Product Page: www.viatrading.com/wholesale-products/category/product-title/ I understand this is better for SEO and user experience. However, we already have good traffic on the current URL Structure. Should we use same left-side filters on Category Pages as in All Products Page? Since we are using Faceted Navigation, when users filter the Category (e.g. Clothing) they will see same page as Clothing Category Page. Is that an issue for Duplicate Content? Since we are a wholesale company - I understand is using "/wholesale/products/" in URL for all product pages a good idea? If so, should we avoid word "wholesale" in product-title to avoid repeated word in URL? For us, SKU in URL helps the company employees and maybe some clients identify the link. However, what do you think of using the SEO-friendly product-title, and 301 redirect it to www.viatrading.com/BRTA-LN-DISHRACKS/, so 1st link is only used by company members and Canonicalized 2nd is the only one seen by general public? Thank you,
On-Page Optimization | | viatrading10 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
How to exclude URL filter searches in robots.txt
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505 How can I exclude all of these filters in the robots.txt? I think it'll be: Disallow: /*?color=$ Is that the correct syntax with the $ sign in it? Thanks!
On-Page Optimization | | neenor0 -
Category page canonical tag
I know this question has been asked a few times on here but I'm looking for very specific advice. Currently when you go to a category, say http://www.bronterose.co.uk/range.html, a canonical tag is added to the head of the page. There are plenty of "variant" pages which carry the same tag, for example: /range.html?p=2
On-Page Optimization | | crichardson9
/range.html?p=3
/range.html?dir=asc&order=price
/range.html?dir=asc&limit=all&order=price Is it wise to push the "link juice" for each of these variant pages to the top level page? Or should each variant page have its own unique canonical tag? After reading many blog posts, guides and papers I'm truly confused! Any general guidance or recommendations would be much appreciated. Chris.1 -
Wordpress: Should I NO INDEX Categories & Archives Pages?
I am new to SEOmoz & trying to work my way through the ca-trillion errors that have been found on my site, but for each one I want to ensure that I am helping rather than harming my site. The tool has (as a "notice") said that my category pages & Archives are NO-INDEX, is this how these pages should be dealt with? In addition, the crawler has also (as a "warning error) discovered that my categories, and Archives do not have a meta description..is this of great importance for non indexed pages of this type? Thanks so much to the SEOmoz forum members, you have so far been of invaluable help to me.
On-Page Optimization | | KMack2