How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Random /feed 404 error from a wordpress site
My Moz Analytics report shows a 404 error on a page which I think should not exist at all. The URL is http://henryplumbingco.com/portfolio-item/butler-elementary/feed/. When I checked webmaster tools, it looks like there are a number of random /feed urls throwing 404 errors. I am using WordPress and the Enfold theme. Anyone know how to get rid of these errors? Thanks,
On-Page Optimization | | aj6130 -
Should I block google indexing "search.php"
My question is I have a search page on our website , you can search by date, number of people staying and so on, I am just wondering should block this in the robots.txt ? Because we have pretty URL'S already for searching by county and searching by towns. I cannot see any benefit of having e.g "search/search.php?sp_dateFrom=16%2F12%2F2015&sp_dateTo=23%2F12%2F2015&sec_drop%5B%5D=727&spesh_town_id=764&q=&occupants=5&bedrooms=3&submit=SEARCH#search" indexed. Would I be correct in doing this ?
On-Page Optimization | | McCaldin0 -
Does a / at the end of a URL create a duplicate page?
Hello, I have just used (the amazing) Screaming Frog to check my site and it is listing the two following pages as having duplicate titles, making me think it is seeing them as duplicate pages. http://zenplugs.com/zenplugs-molded-earphones/ http://zenplugs.com/zenplugs-molded-earphones Do I need to redirect one of these? Thanks in advance! Toby
On-Page Optimization | | T0BY0 -
Magento Canonical & Default Robots Settings
Hello! I'm working with Magento 1.9 for an eCommerce site with several hundred products. Currently I understand it is best practices to use the Canonical tag, however I also have my default robots set to "Index, Follow". Will this cause an issue having product pages set to index, follow but also having a canonical tag included? What are some best practices regarding Magento default robots & canonical tags? Any help is appreciated.
On-Page Optimization | | BretDarby0 -
Keyword repeats/presence in url's & over-optimisation
Hi I'm about to launch a redesigned site and worried about overdoing kw presence on-page, primarily using in url's since will already be using kw in titles as well as page content. What's current thinking re over optimisation: If kw is in titles and page content is it best not to repeat again in url structure i.e. less is more, even though this will cause things like SeoMoz on-page grade score to fall, or better to keep them/add them ? Personally i think it makes sense to include kw in url again since helps make the page relevant, and so long as matches the content should help as opposed to hinder rankings for the pages target keyword. However when i look into this some say don't do this since is over-optimisation The sites generally ranking quite well for its target kw which i obviously don't want to lose after re-launch & hopefully improve further, in the case of this example they are 'Sports Centre Services' & 'Sports Centre Equipment Rental'). The sites current url structure is similar to this below example: frankssportscentres.com/services/sports-centre-equipment-rental Would it be better to keep following existing/above format or to go with either of the below options i.e. more kw rich urls or less: frankssportscentres.com/sports-centre-services/sports-centre-equipment-rental Or frankssportscentres.com/sports-centre-services/equipment-rental Or even less frankssportscentres.com/services/equipment-rental Many Thanks in advance for any helpful comments Cheers Dan
On-Page Optimization | | Dan-Lawrence0 -
Changing an Aged Category Name
I am interested in potentially changing a category name on our site, WirelessEmporium.com. This section has been called "Covers & Faceplates" for the better part of the decade. However, changing search trends, PPC data and search volume analysis have indicated that our preferable KW combo may be "Cases & Covers" instead. What are the potentially ramifications of making the switch now? We have a significant amount of age, link juice, etc built up toward our "Covers & Faceplates" section at the moment and all of their relevant URLs. A change would involve 301 redirects, rewriting of URLs, rewriting of all tags, img alt tags, etc.
On-Page Optimization | | eugeneku0 -
Yahoo Directory! Is it worth Paying $299 for?
Yahoo Directory! Is it worth Paying $299 for? Is this going to help you be better indexed. Will it help your site over all beside having one link?
On-Page Optimization | | AppleCapitalGroup0 -
Sitemap generator and /index.htm
I use DMX Zone sitemap generator in Dreamweaver. When generating the sitemap, it includes both http://www.domain.com and http://www.domain.com/index.htm. Should the /index.htm be deleted from the sitemap, or will the 301 redirect take care of that? Best,
On-Page Optimization | | ChristopherGlaeser
Christopher0