Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Correct robots.txt for WordPress
Hi. So I recently launched a website on WordPress (1 main page and 5 internal pages). The main page got indexed right off the bat, while other pages seem to be blocked by robots.txt. Would you please look at my robots file and tell me what‘s wrong? I wanted to block the contact page, plugin elements, users’ comments (I got a discussion space on every page of my website) and website search section (to prevent duplicate pages from appearing in google search results). Looks like one of the lines is blocking every page after ”/“ from indexing, even though everything seems right. Thank you so much. FzSQkqB.jpg
On-Page Optimization | | AslanBarselinov1 -
Best practice for footer in ecommerce - Shall I add Top Category links?
What would you recommend regarding links to "Top Products" and "Top Categories" in footer? Would you add them to give extra link juice to top categories? would you try to avoid category links in footer that are already in the header navigationor in the main content area to avoid linking twice from all pages? would you vary these top category links in footer according to main category
On-Page Optimization | | lcourse0 -
Category page canonical tag
I know this question has been asked a few times on here but I'm looking for very specific advice. Currently when you go to a category, say http://www.bronterose.co.uk/range.html, a canonical tag is added to the head of the page. There are plenty of "variant" pages which carry the same tag, for example: /range.html?p=2
On-Page Optimization | | crichardson9
/range.html?p=3
/range.html?dir=asc&order=price
/range.html?dir=asc&limit=all&order=price Is it wise to push the "link juice" for each of these variant pages to the top level page? Or should each variant page have its own unique canonical tag? After reading many blog posts, guides and papers I'm truly confused! Any general guidance or recommendations would be much appreciated. Chris.1 -
Canonical URL, cornerstone page and categories
If I want to have a cornerstone "page", can I substitute an actual page with a category archive of posts "page" (that contains many posts containing the target key phrase)? This way, if I make blog posts about a certain topic/ key phrase (example "beach weddings") and add a canonical URL of the category archive page to the individual posts, am I right then to assume google will see the archive page as the cornerstone page (and thereby won't see the individual posts with the same key phrase as competing)?
On-Page Optimization | | stephanwb0 -
Does Google index dynamically generated content/headers, etc.?
To avoid dupe content, we are moving away from a model where we have 30,000 pages, each with a separate URL that looks like /prices/<product-name>/<city><state>, often with dupe content because the product overlaps from city to city, and it's hard to keep 30,000 pages unique, where sometimes the only distinction is the price & the city/state.</state></city></product-name> We are moving to a model with around 300 unique pages, where some of the info that used to be in the url will move to the page itself (headers, etc.) to cut down on dupe content on those unique 300 pages. My question is this. If we have 300 unique-content pages with unique URL's, and we then put some dynamic info (year, city, state) into the page itself, will Google index this dynamic content? The question behind this one is, how do we continue to rank for searches for that product in the city-state being searched without having that info in the URL? Any best practices we should know about?
On-Page Optimization | | editabletext0 -
How to avoid keyword stuffing on e-Commerce Category pages
Hi, I'm optimizing a large, consumer electronic e-commerce superstore. Based on client's choice of keywords, I'm using product category pages as my target urls. Because of the proprietary CMS structure, product names and titles, featured on my landing pages (product category pages) create a keyword overkill, affecting various ranking factors. For example, one of the target urls / landing pages, dedicated to a specific product category, mentions the keyword over 190 times because of so many product titles in the "body" section. Would inline "rel="canonical" help? If yes, what part of the website should it "canonize"? If rel="canonical" is not the answer, what strategies would you suggest? Thanks!
On-Page Optimization | | dimanyc0 -
Title tag for category page
I'd like to know your views on the best approach for title tags for category pages for ecommerce sites. 3 examples A) Category name | Free delivery on $50 purchase | Brand name B) Discover best "category name" on brand name C) Category Name | 1st Keyword, 2nd keyword | Brand name Thanks!
On-Page Optimization | | walidalsaqqaf0 -
Google Page Rank of my site has dropped from 4/10 to 3/10
Google Page rank of my website has been dropped after Panda Update. Can anyone help me out to tell me the possible reasons about the same. We have tried to make our website more lively and user friendly. We have indulged some graphics to make it more attractive. But it seems it backfired us. my site is http://www.myrealdata.com as well as Google page ranking of my Quickbooks hosting page has been dropped as well. It would be great if someone can help me out with expert suggestions.
On-Page Optimization | | SangeetaC1