How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Hi - How do you get rid of duplicate content that was accidentally created on a tag url? For example, when I published a new article, the content was duplicated on: /posts/tag/lead-generation/
the original article was created with: /posts/shippers-looking-for-freight-brokers/ How can I fix this so a new URL is not created every time I add a tag to a new posting?
On-Page Optimization | | treetopgrowthstrategy0 -
Are there detrimental effects of having multiple robot tags
Hi All, I came across some pages on our site that have multiple robot tags, but they have the same directives. Two are identical while one is for Google only. I know there aren't any real benefits from having it set up this way, but are there any detrimental effects such as slowing down the bots crawling these pages? name="googlebot" content="index, follow, noodp"/> Thanks!
On-Page Optimization | | STP_SEO0 -
URL Structure Category Pages -Current Moz Friday-
Hello,
On-Page Optimization | | _Heiko_
regarding #15 of the last moz friday I have a question: http://moz.com/blog/15-seo-best-practices-for-structuring-urls What would you prefer if the lenght of the URL will be still under 60 characters and you have an example like this: Let's call it a specific page in a category. As I like the old shoe examples: You have a page about red shoes in your shoe category. Which URL would you prefer: a) www.mydomain.com/shoes/red-shoe b) www.mydomain.com/shoes/red Personally I would prefer a) or would you already consider this as spammy? My real example is not that trivial like the shoe example and the categories will be in plural and the specific pages always in singular (like in the example shoes vs shoe). c) would be to put it independently from the side structure on www.mydomain.com/red-shoe - but personally I have the experience that a) or b) will help the rankings of the category page if you have the specific pages in the same subfolder. What's your opinion on this?1 -
Removing entire site sections categories from an ecommerce store, best practices ?
Hi Whats best practice when removing entire site sections from an e-commerce store due to ending those lines and not having similar alternative pages to redirect them too ? Should you remove pages in GWT etc ? Any other measures to prevent probs such as 404 spikes etc ?? Cheers Dan
On-Page Optimization | | Dan-Lawrence0 -
Wordpress photo blog with sparse text - noindex posts, index categories?
Hi everyone, I have a wordpress blog that is heavy on photos but short on text - most of my posts consist of a photo linking to a full gallery, and a short description of 1-4 sentences. I've often read recommendations to noindex category pages, but in this case I'm wondering if it might be best to noindex the individual posts instead - I'm concerned that the short posts might seem like thin content to Google compared to the category pages which aggregate the posts. Also, some of my categories reflect keywords that I'd like to rank well for. I have about a dozen categories and close to 1000 posts.
On-Page Optimization | | matt_b0 -
Domain sub-directory not performing
We've restructured our site over the past 6 months and I'm going to run you through the whole scenario as I would love any feedback you guys have. 6 months ago we had 2 websites http://boulders-climbing.com (climbing facility) and http://bouldersuk.com (shop). We made the decision to merge the websites and leverage the SEO on 1 site. Although boulders-climbing.com was the older and more established domain, the company wanted to use bouldersuk.com so a whole new website was designed and boulders-climbing.com was redirected to bouldersuk.com. The climbing facility website now sits at bouldersuk.com and the shop was moved to bouldersuk.com/climbing-shop with 301's for all shop pages. This has lead to a significant a increase in domain rank for bouldersuk.com and much better rankings for the climbing centre related terms. The desired effect has been achieved, well half of it anyway. The search rankings for bouldersuk.com/climbing-shop have never reached the previous heights and are still heading in the wrong direction, even though the overall domain ranking has increased by 50%. What can I do to get the SEO for /climbing-shop working again? We're adding fresh content to our latest news with links through to products and categories, all category pages have A grades. we are attempting to link build but it is much more difficult for e-commerce than for the facilities pages. Is the SEO of the main site hampering(masking?) the /climbing-shop? All feedback on the whole process would be much appreciated. Thanks!
On-Page Optimization | | benj450 -
Blog.mysite.com or mysite.com/blog?
Hi, I'm just curious what the majority think of what's the best way to start a blog on your website for SEO benefits. Is it better to have it under a sub domain or a directory? Or does it even matter?
On-Page Optimization | | truckguy770 -
Seomoz.org Category and Tags practice
Hello, I have been checking seomoz sourcecode and architecture these days in order to learn and to apply it in my site but I havent managed to find "tags" at all. Just the "Posts by Categories" on the right sidebar. Is this the only way you are categorising content? In this case, the only way spiders have to find your content is via these category archive pages and the general sitemap? Thanks!
On-Page Optimization | | antorome0