How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Recommend platform for fashion/bridal store
We currently use Bigcommerce, can anyone recommend a good platform for a bridal fashion store. We use a lot of pictures and banners. Also looking for a platform that allows for good SEO We are looking for an updated look particularly with a responsive format for mobile phones and ipads.
On-Page Optimization | | CostumeD0 -
Any idea how Google is doing this? Is it schematic? http://techcrunch.com/2014/02/28/google-adds-full-restaurant-menus-to-its-search-results-pages/
Google is now showing menus on select searches. Any idea how they are getting this information? I would like to make sure my clients get visibility this way.
On-Page Optimization | | Ron_McCabe0 -
The effect of having CR LF HT commands in a <title>tag</title>
Hello. I am looking at a customer site with a CMS system that is controlling the population of the meta TITLE Currently it has the TITLE set as this <title>(CR)(LF)<br />(HT)Site Details REMOVED - Customer name REMOVED(CR)(LF)<br /></title> Naturally, we would prefer it to be <title>Site Details REMOVED - Customer name REMOVED</title> What affect would these commands have in the title! Google shows their title when you Google the company website... so I guess it can see it .... but GA "Top Site Content" widget shows it as blank ? Any ideas? Cheers
On-Page Optimization | | BinaryTris0 -
Changing an Aged Category Name
I am interested in potentially changing a category name on our site, WirelessEmporium.com. This section has been called "Covers & Faceplates" for the better part of the decade. However, changing search trends, PPC data and search volume analysis have indicated that our preferable KW combo may be "Cases & Covers" instead. What are the potentially ramifications of making the switch now? We have a significant amount of age, link juice, etc built up toward our "Covers & Faceplates" section at the moment and all of their relevant URLs. A change would involve 301 redirects, rewriting of URLs, rewriting of all tags, img alt tags, etc.
On-Page Optimization | | eugeneku0 -
Re-targeting keywords from homepage to category page
My website home page ranks well for a keyword lets call it 'Keyword1'. I'd like to create a new category specific to 'Keyword1' and name it 'Keyword1' so the relevant content and products can sit in there. How can I ensure that my rankings of my current home page are passed on to this new category page? Many thanks,
On-Page Optimization | | alexhowe0 -
Recommended SEO company/expert
It's as simple as the title. We're a well established ecommerce company about to move away from the actinic platform on to a new magento site. Of late our long held rankings have slipped and we are looking for guidance in making up the lost ground and getting the most SEO wise from magento. Thanks is advance.
On-Page Optimization | | LadyApollo0 -
How do you create a 301 redirect for www.mysite..com/index.html in htaccess.
I understand that it is possible to create a 301 redirect for www.mysite..com/index.html to www.mysite.com. (as well as subdirectories.) How is this accomplished? My hosting company says that setting this up in htaccess will cause "Apache to geti into an infinite loop and the page won’t load." I have read on the forum that this is possible. Any help would be greatly appreaciated. THanks. Perri
On-Page Optimization | | PerriCline0 -
100 Words per category
These are the numbers of words I have heard mentioned as being "good enough" for the categories of ecommerce pages. Even at a stretch 50 words maximum. What are peoples opinions on the volume of words required ?
On-Page Optimization | | onlinemediadirect0