How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Use H2 or H3 for Six Ways to Hip Thrust https://www.fitness-china.com/hip-thrust
Our article about Hip Thrust https://www.fitness-china.com/hip-thrust We have sex tips about how to hip thrust. that use h2. but Final Words use h3 or h2?
On-Page Optimization | | ahislop5740 -
Category Pages are not Shown when searched for "brandname Category name"
Hi, Someone googled our brand name + category page name and our category page didn't showed up first so he says it's a bad job/thing. so help me to fix this. Googled "sepalika acid reflux" without quotes. Thank you.
On-Page Optimization | | Sepalika0 -
Two Robots.txt files
Hi there Can somebody please help me that one of my client site have two robot.txt files (please see below). One txt file is blocked few folders and another one is blocked completely all the Search engines. Our tech team telling that due to some technical reasons they using second one which placed in inside the server and search engines unable to see this file. www.example.co.uk/robots.txt - Blocked few folderswww.example.co.uk/Robots.txt - Blocked all Search Engines I hope someone can give me the help I need in this one. Thanks in advance! Cheers,
On-Page Optimization | | TrulyTravel
Satla0 -
Robot.txt file issue on wordpress site.
I m facing the issue with robot.txt file on my blog. Two weeks ago i done some development work on my blog. I just added few pages in robot file. Now my complete site seems to be blocked. I have checked and update the file and still having issue. The search result shows that "A description for this result is not available because of this site's robots.txt – learn more." Any suggestion to over come with this issue
On-Page Optimization | | Mustansar0 -
Directories and Search Engine Site Submissions and Technorati Tags Important?
I use a wordpress for out website 50campfires.com and we have a site map set up along with google analytics and web master tools. I believe wordpress automatically sends information to the search engines with changes in the site map. Do I need to submit to search engines manually once a month? Is directories like DMOZ still relevant with SEO today? And what are your opinions on Technorati Tags? I never heard of them before reading this article. I m new to SEO and any insights are helpful. 2Sqv5C2.png EEQFpIx.png 2Sqv5C2.png
On-Page Optimization | | revonick1 -
Indexed/Submitted URLS vs Total Indexed
Hello, My site is www.colbysphotography.com. I have Total Indexed 195 while I have 87 URLs submitted and only 79 URLs Indexed. What is the difference and is there a problem? Thanks ahead of time,
On-Page Optimization | | littlecolby
Colby0 -
Different Categories, Same Meta Description?
I've heard that having duplicate meta descriptions is bad for SEO and can even be the cause of penalties. However, a duplicate meta description would make sense for different categories on many sites. So I'm wondering what the best solution is for a case like this? One of my sites has around 150 topics, and each topic page is exactly the same, aside from including only things that are related to that specific topic. So why would I want to create 150 different meta descriptions? Not only is it time consuming, and nearly impossible to create 150 unique descriptions for the same type of content, but it also serves no purpose for visitors coming from the search engines. A logical approach would be to use the same meta description and just switch out the topics in the description... but then all of those descriptions would be 95% the same - and I imagine Google might see that as spam. So... suggestions?
On-Page Optimization | | JABacchetta0 -
Different names for Twitter/Facebook, and site?
Greetings, I want to optimize my site, www.atstrust.com. I know that posts to Facebook and Twitter help improve ranks. Does it matter though if my twitter/facebook sites have a different name than the domain I am optimizing? For example, my Facebook site is facebook.com/easyinspection. I assume that doesn't matter as long as the URLs on the site refer to my domain? Thanks! Eric
On-Page Optimization | | Ericc220