Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Schema Markup for eCommerce Category Pages?
My research indicates that applying an ItemList schema markup to our category pages is likely the best way to go. However, I've also ready that Google discourages schema markup on category pages. I'm just wondering if any of you have applied schema markup to your category pages and, if so, how did you do it? John,
On-Page Optimization | | JohnBrown75
Essay Writer1 -
Using h2 for category on ecommerce website
Hi, I am working on an ecommerce site and the main category - lets call them car widgets - is using a h1 at the top of the page which is great. There are 4 sub categories on the page - lets call one of them red widget. The only content on the page is the sub category name and an image. Should the sub category red widget use a h2? Thanks S
On-Page Optimization | | bedynamic0 -
Should we rename and update a page or create a new page entirely?
Hi Moz Peoples! We have a small site with a simple site navigation, with only a few links on the nav bar. We have been doing some work to create a new page, which will eventually replace one of the links on the nav bar. The question we are having is, is it better to rename the existing page and replace its content and then wait for the great indexer to do its thing, or perm delete the page and replace it with the new page and content? Or is this a case where it really makes no difference as long as the redirects are set up correctly?
On-Page Optimization | | Parker8180 -
Duplicate content with tagging and categories
Hello, Moz is showing that a site has duplicate content - which appears to be because of tags and categories. It is a relatively new site, with only a few blog publications so far. This means that the same articles are displayed under a number of different tags and categories... Is this something I should worry about, or just wait until I have more content? The 'tag' and 'category' pages are not really pages I would expect or aim for anyone to find in google results anyway. Would be glad to here any advice / opinions on this Thanks!
On-Page Optimization | | wearehappymedia1 -
Harms of hidden categories on SEO
On our website we have some invisible/hidden categories on our site. Can anyone advise whether these are harmful in terms of SEO?
On-Page Optimization | | CostumeD0 -
Can you use the canonical tag and rel=next and rel=prev on category pages.
We have a conflict of information between our web developers and our SEO company. We are an on-line retail company hence we have a fair number of different categories. Our site is set up with the rel=next and rel=prev tags. Our SEO company have asked us to implement canonical links on our category pages and leave the rel=next and rel=prev tags as they are. Our web developers are saying by doing this we are asking Google to ignore all of our products on all of the pages except page 1 which would mean Google would not index a lot of our products. I have looked at a few articles but I am struggling to understand which way to go. Any advice would be appreciated. Thank you in advance.
On-Page Optimization | | Palmbourne0 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
Ecommerce - how many clicks from the home page should categories be
My client has about 300 products in 20 categories with a lot of overlap. How many clicks from the home page should we keep the products? We're not doing pagination. I'd been told several years ago that all products should be 2 clicks or less from the home page. Is this true today? Thanks.
On-Page Optimization | | BobGW1