How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Aggregator/comparitor site outranking us
Hi I would like to know if anyone has experience with trying to outrank an aggregator/comparitor website. We are being beat by one that also includes our range of products in their comparisons and I was wondering if there was a smart way around this?
On-Page Optimization | | Discovery_SA2 -
Page/Website Structure
Hello again Mozzers, We have a category, lets call it widgets. Within widgets are about a hundred or so products. For usability my predecessor made the following layout Widgets Main Cateogry - Links off homepage - (no content just links to the 3 sub-categories)
On-Page Optimization | | ATP
- Widgets by Resolution
---- About 20 subcategories
eg. 0.1 Resolution widgets
0.2 resolution widgets
- Widgets by Capacity
---- About 20 subcategories
eg. 1 capacity widgets
2 capacity widgets
- Widgets by Type
---- About 12 subcategories This was a major improvement from a userbility perspective as it made a very complex product range navigatable by the major features or basic type. However, as you can imaging we now have 60+ very similiar pages all displaying very similiar products a nightmare for SEO. It also isnt ideal for user navigation as it take too many clicks to get to the products. I propose the following fix, and i wanted your opinion. Widget Main Category - Link from homepage (Consolidated with Widgets by Type)
-300 Words of content
-Links to the 12 Sub-type Catoregies (These are pages i can fill with content + products. This would give me a more ordinary structure of which I can focus each page to a keyword) The tricky part comes with incorporating the capacity and resolution options. 1 Browse Capacity Page
(20 sub categories all the same except capacity quantity & products)
1 Browse by Resolution Page
(20 sub categories all the same except resolution value & products) The owner want them, I was going to link from the main widgets page to each of these to give the customer the option. What I can't decide is how to deal with them from an SEO point of view. Should they be no-followed? canonicaled? Can there be any advantage to having so many pages covering slightly different variations or as i suspect it is dangerous to the overall health of the site. To complicate things further, Canonical tags may not be an option due to an old magento version running that doesnt support them. Is there an alternative way around? As always many thanks.0 -
Ratio follow/nofollow outbound links
Dear all, So far, I couldn't find any satisfying answer to my problem - I hope you might be able to help: Due to the fact that our website consists of user-generated content, we've got many many outbound links to other sites. Until now, it was possible to assign a follow-attribute to these links. In the ages of "pandas" however, we realised we had to limit this possibility as well as the amount of outbound follow links already published. My question now is whether there is some kind of rule of thumb as to what ratio of outbound follow and nofollow links is advisable. I'd appreciate any ideas or comments. Thanks a lot!
On-Page Optimization | | Mulle0 -
Link juice/PR to social media platforms
Hi, Do you know if Google gives as much link juice/PR to social media platforms (when linking to them from a website), as the give to perhaps another internal/external link? The thought is: As Google is getting more intelligent and gives social media that much value in SERP, would the pass on/through link juice to the social media. Hope my question is understandable. Thks.
On-Page Optimization | | teambenns_seo0 -
It is NOW time to invest in SEO consultant, How/ Where can i find a good one???
I have learn the basics of seo and use seoMoz for more then a year. this help me to rank my online store better and increase sales. I only know the basics of seo and it is now time increase traffics to our online store, www.furnacefilterscanada.com How can I hired a good seo consultant? I'm self employes and the only one managing my online store. I want to be shure I invest in a good consultant. Our small business does not have thousand$$$$$ dollards to invest. For example, can I see result if I invest $1000? Thank you
On-Page Optimization | | BigBlaze2050 -
Ecommerce - how many clicks from the home page should categories be
My client has about 300 products in 20 categories with a lot of overlap. How many clicks from the home page should we keep the products? We're not doing pagination. I'd been told several years ago that all products should be 2 clicks or less from the home page. Is this true today? Thanks.
On-Page Optimization | | BobGW1 -
Which is better, a directory 301 redirect or each page in the directory?
A customer of mine has a site with lots of articles and they are all quite spammy. They have not been affected by penguin yet so they asked what to do. I suggested losing the articles directory and 301 redirect to either the home page or another important page. Would a 301 redirect on the entire directory to a single page be the way to go or add redirects from each page within the directory and spread out redirects to various pages in website? Or do you have a better suggestion?
On-Page Optimization | | anthonytjm0 -
Duplicate page content & title for www.mydomain.com and www.mydomain.com/index.php?
Hi, First post so please be gentle! My Crawl Diagnostics Summary is showing an error relating to duplicate page content and duplicate page title for www.mydomain.com and www.mydomain.com/index.php which are, in my view, the same thing/page? Could anyone shed any light please? Thanks Carl
On-Page Optimization | | Carl2870