Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How do i block an entire category/directory with robots.txt?
-
Anyone has any idea how to block an entire product category, including all the products in that category using the robots.txt file? I'm using woocommerce in wordpress and i'd like to prevent bots from crawling every single one of products urls for now.
The confusing part right now is that i have several different url structures linking to every single one of my products for example www.mystore.com/all-products, www.mystore.com/product-category, etc etc.
I'm not really sure how i'd type it into the robots.txt file, or where to place the file.
any help would be appreciated thanks
-
Thanks for the detailed answer, i will give it a try!
-
Hi
This should do it, you place the robots.txt in the root directory of your site.
User-agent: * Disallow: /product-category/
You can check out some more examples here: http://www.seomoz.org/learn-seo/robotstxt
As for the multiple urls linking to the same pages, you will just need to check all possible variants and make sure you have them covered in the robots.txt file.
Google webmaster tools has a page where you can use to check if the robots.txt file is doing what you expect it to do (under Health -> Blocked Urls).
It might be easier to block the pages with a meta tag as described in the link above if you are running a plugin allowing this, that should take care of all the different url structures also.
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Value of using spaces or no spaces on product category page varient keywords
Hello, all fellow Mozzers,
On-Page Optimization | | JamesDavison
I have taken over a project and this account, so can't change the username according to MOZ.🙃 We run an eCommerce website, and to me, some of the content is conflicting as some pages have more information content than what I would put in a commerce page, but this is how the boss wants it to work, personally, I would separate the content out.
The page I'm working on:
https://www.longstonetyres.co.uk/tyres/205-70-14.html
and this is an example of the rest of these types of pages, I will be tackling:
https://www.longstonetyres.co.uk/tyres/125-15.html I was tasked to improve SEO ranking, when using the MOZ page grader I had a score of 24 out of 27 83% SEO score and 3-page problems. 7th position in Google for the search term 205/70 R14 As it is a generic product listing page, It was pointless to add to the URL and the Internal links I can't reduce as these are links to products, so I went to reduce the
keyword stuffing and making the page content more natural, this improved the page to 25 out of 27, 87% SEO score and 2-page problems. Improvement to 3rd position in Google, but he wants to chase 1st place to be above his competitors, which is fair enough. It turns out that in the past, they have used this type of page to try and get a high ranking for several search terms, as it is a different variation on a tyre size terms are:
205/70 R14, 205/70R14, 205/70 R 14
205/70 X 14, 205/70X14, 205/70 X14
and so on for all the different ways you can search for this tyre size. He is also convinced Google will see these as different search terms, and while I agree to an extent, this causes Keyword Stuffing on the page, which in turn was harming the rankings. Each product listed on the page already has its own title 205/70 R14, 205/70 HR14 and so on, so my question is. What is the best practice for writing content on these types of pages to gain high rankings for several Keywords, and what value does writing the same keyword with spaces and no spaces have? Any help or advice is welcome, so I have a better understanding of how to approach this for this page and the rest of the site. Cheers Mal0 -
Updating Old Content at Scale - Any Danger from a Google Penalty/Spam Perspective?
We've read a lot about the power of updating old content (making it more relevant for today, finding other ways to add value to it) and republishing (Here I mean changing the publish date from the original publish date to today's date - not publishing on other sites). I'm wondering if there is any danger of doing this at scale (designating a few months out of the year where we don't publish brand-new content but instead focus on taking our old blog posts, updating them, and changing the publish date - ~15 posts/month). We have a huge archive of old posts we believe we can add value to and publish anew to benefit our community/organic traffic visitors. It seems like we could add a lot of value to readers by doing this, but I'm a little worried this might somehow be seen by Google as manipulative/spammy/something that could otherwise get us in trouble. Does anyone have experience doing this or have thoughts on whether this might somehow be dangerous to do? Thanks Moz community!
On-Page Optimization | | paulz9990 -
Best schema option for condos / condominiums?
Hey guys, I'm doing a review on some schema on some of our sites. Most of them are generic using LocalBusiness. There are a few more specific schemas I could use, but not sure what would be the most relevant. Wondering if any of you have a suggestion or ideas? https://schema.org/Residence https://schema.org/LodgingBusiness https://schema.org/ApartmentComplex or I could just stick with LocalBusiness. I'm leaning towards LodgingBusiness or ApartmentComplex.... but when I think of LodgingBusiness I think of something temporary / vacation type deal like hotels. Apartments... kind of self explanatory, a condominium isn't exactly an apartment but perhaps it is more comparable to an apartment than a hotel, motel or inn. What are you thoughts on this? Also, which "format" is better to use RDFa, microdata, or JSON-LD. Does it matter?
On-Page Optimization | | donnieath0 -
Using keywords in my URL: Doing a redirect to /keyword
My website in "On Page Grade" received an A.Anyway, I only have 1 thing to optimize:_"Use Keywords in your URL__Using your targeted keywords in the URL string adds relevancy to your page for search engine rankings, assists potential visitors identify the topic of your page from the URL, and provides SEO value when used as the anchor text of referring links."_My website is ranking in top10 for a super high competitive keyword and all my others competitors have the keyword on their domain, but not for my URL.Since I can't change my domain for fixing this suggestion, I would like to know what do you think about doing a 301 redirect from / to mydomainname.com/keyword/So the index of my website would be the /keyword.I don't know if this can make a damage to my SERP for the big change ir it would be a great choice.
On-Page Optimization | | estebanseo0 -
Alt text / internal linking
Hi everyone A question about best practice when linking from pictures on our homepage - hirespace.com We have an option of using divs with background images (nicer in terms of design) but it means that we can't use anchor text or alt text to show Google what these internal links are about. The other option is to use images which do not allow us as much flexibility in terms of CSS but would allow us to use alt text. There is also an opinion that we should have separate text links at the bottom of the homepage to get the anchor page in. What is best practice in this situation - is alt text worth sacrificing some CSS flexibility for? How important is anchor/alt text for internal linking? Thanks guys.
On-Page Optimization | | HireSpace0 -
301 Redirect to product page or category?
We manage an ecommerce website that sells health products. A few products have now been discontinued. I’m just wondering what would be the best practice in this case. Should we 301 redirect to a similar product or to a similar category page? ANY HELP IS GREATLY APPRECIATED!
On-Page Optimization | | odegi0 -
How much copy should there be on an e-commerce category page?
I'm not looking for a precise number, obviously. I'm more interested in a general range. More text means more long-tail and synonym opportunities, but of course you don't want too much copy above the fold, pushing your products down. Maybe you can get away with a short paragraph or two at the top of the page. You can always put more copy below the products, but in a recent SEOmoz e-commerce webinar, the presenter seemed to think that was silly and unnecessary. He even suggested that the algo might intentionally ignore text below products, since it's clearly not intended to be read. What do you think?
On-Page Optimization | | CMC-SD0 -
Wordpress: Should I NO INDEX Categories & Archives Pages?
I am new to SEOmoz & trying to work my way through the ca-trillion errors that have been found on my site, but for each one I want to ensure that I am helping rather than harming my site. The tool has (as a "notice") said that my category pages & Archives are NO-INDEX, is this how these pages should be dealt with? In addition, the crawler has also (as a "warning error) discovered that my categories, and Archives do not have a meta description..is this of great importance for non indexed pages of this type? Thanks so much to the SEOmoz forum members, you have so far been of invaluable help to me.
On-Page Optimization | | KMack2