Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Correct robots.txt for WordPress
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong? I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right. Thank you so much. FzSQkqB.jpg
On-Page Optimization | | AslanBarselinov1 -
Impact of keyword/keyphrases density on header/footer
Hi, It might be a stupid question but I prefer to clear things out if it's not a problem: Today I've seen a website where visitors are prompted no less than 5 times per page to "call [their] consultants".
On-Page Optimization | | GhillC
This appears twice on the header, once on the side bar (mouse over pop up), once in the body of most of the pages and once in the footer. So obviously, besides the body of the pages, it appears at least 4 times on every single pages as it's part of the website template. In the past, I never really wondered re the menu, the footer etc as it's usually not hammering the same stuff repeatedly everywhere. Anyway, I then had a look at their blog and, given the average length of their articles, the keyword density around these prompts is about 0.5% to 0.8% for each page. This is huge! So basically my question is as follow: is Google's algorithm smart enough to understand what this is and make abstraction of this "content" to focus on the body of the pages (probably simply focusing on the tags)? Or does it send wrong signals and confuse search engine more than anything else? Reading stuff such as this, I wonder how does it work when this is not navigational or links elements. Thanks,
G Note: I’m purposely not speaking about the UX which is obviously impacted by such a hammering process.0 -
Updating Old Content at Scale - Any Danger from a Google Penalty/Spam Perspective?
We've read a lot about the power of updating old content (making it more relevant for today, finding other ways to add value to it) and republishing (Here I mean changing the publish date from the original publish date to today's date - not publishing on other sites). I'm wondering if there is any danger of doing this at scale (designating a few months out of the year where we don't publish brand-new content but instead focus on taking our old blog posts, updating them, and changing the publish date - ~15 posts/month). We have a huge archive of old posts we believe we can add value to and publish anew to benefit our community/organic traffic visitors. It seems like we could add a lot of value to readers by doing this, but I'm a little worried this might somehow be seen by Google as manipulative/spammy/something that could otherwise get us in trouble. Does anyone have experience doing this or have thoughts on whether this might somehow be dangerous to do? Thanks Moz community!
On-Page Optimization | | paulz9990 -
Removing navigation menu items/links on homepage
We are redesigning our website after a long stint with an SEO firm who also handled our design/dev. We want to clean up the links on our homepage but don't want to screw up our IA or SEO. We want to delete some navbar menu items and a whole bunch on random links to our evergreen content below the fold. Would we need to reposition those navbar items/content links to our footer or somewhere else on the homepage to maintain our internal linking structure? It would be great if you could take a look at our site and give us any suggestions or advice on the best way to go about this. Thanks!
On-Page Optimization | | Lorne_Marr1 -
H2s & H3s for Category Navigation
Hi all. I am wondering how best to format a category navigation menu. Currently I don't think we're using H2s correctly on our website. Am I right to think that the top level category e.g. Games should be formatted as an H2 and the sub-categories underneath this should be formatted as H3s (to show a hierarchy)? Is there a limit on how many H2s and H3s you should use? Obviously only one H1 per page. Thanks in advance Paul
On-Page Optimization | | kevinliao0 -
Blocking Subdomain from Google Crawl and Index
Hey everybody, how is it going? I have a simple question, that i need answered. I have a main domain, lets call it domain.com. Recently our company will launch a series of promotions for which we will use cname subdomains, i.e try.domain.com, or buy.domain.com. They will serve a commercial objective, nothing more. What is the best way to block such domains from being indexed in Google, also from counting as a subdomain from the domain.com. Robots.txt, No-follow, etc? Hope to hear from you, Best Regards,
On-Page Optimization | | JesusD3 -
Canonical URL, cornerstone page and categories
If I want to have a cornerstone "page", can I substitute an actual page with a category archive of posts "page" (that contains many posts containing the target key phrase)? This way, if I make blog posts about a certain topic/ key phrase (example "beach weddings") and add a canonical URL of the category archive page to the individual posts, am I right then to assume google will see the archive page as the cornerstone page (and thereby won't see the individual posts with the same key phrase as competing)?
On-Page Optimization | | stephanwb0 -
Http://www.xxxx.com does not re-direct to http://xxx.com
When typing in my website URL www.earthsaverequipment.com successfully re-directs to earthsaverequipment.com as specified in robot. However if you type http://www.earthsaverequipment.com it brings up a 404 error Is this a potential issue? if so is there a way to fix it? thanks
On-Page Optimization | | Earthsaver0