Layered navigation and hiding nav from user agent
-
I am trying to deal with the duplicate content issues presented by Magento's layered navigation feature (aka faceted navigation). I installed Amasty's Improved Navigation extension (https://amasty.com/improved-layered-navigation.html) and it offers the option to hide the layered navigation from specific user agents (ie googlebot, bingbot, etc).
This seems like cloaking to me and I hesitate to try it, unless hiding faceted navigation from specific user agents is known to be acceptable to Google (white hat practice). Does anyone know if this the case?
-
Great, thanks Carson! You're insights have been very helpful. I think we'll try to make the on-page ajax solution work.
-
If you're really worried about indexation I think that's a fine solution. It's definitely easier to manage, and it'll also be easier to track pageviews in most analytics platforms. The only downside is that if someone emails or links to a category page with filters applied the recipient won't see it. But generally people share products and not category pages, so it's not a big deal. I'd probably go that route.
Also make sure that your category pages still update the URL when you go to page 2, or that page 2 is somehow also being indexed. You don't want products that don't get indexed because categories can't be crawled.
-
Thanks for the link! I can see how Google offers me a way to tell it how to use my site variables. It seems like between managing parameters in webmaster tools, using canonical links and adding meta noindex tags on variable pages, there should be enough to keep things in order with the search engines. And I can just assume Google knows not to waste too much crawl budget on the variable pages.
I was considering one other option that would remove concerns about variables altogether. Using a different extension, I can set up Magento's layered navigation to work on the page without updating the URL. This eliminates the need for canonicals, parameters, and everything else that is more in Google's control than mine. What do you think of that as a solution?
-
Yes, the bots will crawl the pages, but they will not INDEX them.
There is a concern there, but mostly if the bots get caught in some kind of crawl trap - where they're trying out a near-infinite set of variables and getting stuck in a loop. Otherwise the spiders should understand the variables. You can actually check it in Webmaster tools to make sure Google understands. Instructions for that here:
https://support.google.com/webmasters/answer/6080550?hl=en
Ultimately Google will definitely not penalize you for having lots of duplicate content on URLs through variables, but it might be an issue with Googlebot not finding all your pages. You can make sure that doesn't happen by checking the indexation of your sitemap.
You could also try to block any URLs with the URL parameter in robots.txt. Make sure you get some help on the RegEx if you plan to do this. My advice is that blocking the variables in robots.txt is not worth it, as Google should have no problems with the variables - especially if the canonical tags are working.
Googlebot at least is smart enough these days to know when to stop crawling variable pages, so I think there are more important on-site things to worry about. Make sure your categories are linked to and optimized, for example.
-
This gets into an issue of bots and crawling where I am less clear. Even with canonicals, don't search engine bots crawl all of the pages produced with faceted navigation? That will easily reach 10,000+ pages on my site, which currently has a total number of pages in the low hundreds. I was under the impression I don't want to set up the faceted navigation in a way where the bots crawl through every combination of pages created by my products' attribute filters and bog the bots down in a quagmire of low-value pages. But I'm not sure if that's the case or how concerned I need to be about the bots spending their time on those pages.
-
If I'm not mistaken Magento has canonical tags on category pages by default, so you might be trying to solve an issue that doesn't exist. Take a look at the source code on faceted navigation to confirm. Or you can send me the site and I'll look over it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to find Topics ? software or results and user intent
Hello, Is there a software that is better than an other to find to right topics to cover in my content. I am thinking about Moz, Marketmuse or Semrush or is it better to look at the search results because they match user intent and see what is covered and cover those in my content Thank you,
Intermediate & Advanced SEO | | seoanalytics1 -
Hide Aggregation from Google?
Google isn't a fan of aggregation, but sometimes it is a good way to fill out content when you cannot cover every news story there is. What I'm wondering is if anyone has continued to do any form of aggregation based on a category and hide that url from Google. Example: example.com/industry-news/ -- is where you'd post aggregation stories but you block robots from crawling that. We wouldn't be doing this for search value just value to our readers. Thoughts?
Intermediate & Advanced SEO | | meistermedia0 -
What Navigation strategy should I pursue for this local SEO project? Why?
Hi Mozzers, Context: I am working with a plumbing company that is located in NC and covers 6 locations( 1 address + 5 target cities). To start off I am planning to SEO his main location + 2 to 3 areas. The initial plan is to create 30 (10 services for 3 locations) unique landing pages for the main area and the extra locations,. His services are: PLUMBING EMERGENCY PLUMBING WATER TREATMENT WATER HEATER INSTALLATION WATER HEATER REPAIR TANKLESS WATER HEATERS SEWER AND DRAIN CLEANING REPIPING GARBAGE DISPOSALS GENERAL PLUMBING Since I am including 2 to 3 extra areas offering all these services above, should I include a subnavigation which will create a localized microsite within the site itself for these extra locations or not? Please specify why? Thanks!
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Is a dynamic online user list bad for SEO?
Hello everyone, I have a question that is currently puzzling me, and I hope you can help me with. On musicianspage.com (one of our websites), we show a list of online users embedded within the page which, as you may expect, changes all the time according to who's online at that moment. That list appears on every page of the site, so at any time any page on the site has a different content and different link profile (sometimes we have just a few users connected, other times we may have over 50 users connected at the same time). My question is: is such a "dynamical-embedded" list bad, good or neutral from a SEO stand point? If it is bad, what do you suggest to do? Put it inside a frame? Using AJAX? Any thoughts and suggestions are very welcome! Thanks in advance to anyone reading this. All the best, Fabrizio
Intermediate & Advanced SEO | | fablau0 -
HTML5 Nav Tag Issue - Be Aware
In checking my internal links with GWT, it is apparent that links within the nav tag in HTML5 are discounted by Google as "internal links" This could have major repercussions for designing your internal link structure for SEO purposes. I was surprised to see this result, as I have never seen it discussed. Anyone else notice this, or have any alternative views?
Intermediate & Advanced SEO | | veezer0 -
Techniques to fix eCommerce faceted navigation
Hi everyone, I've read a lot about different techniques to fix duplicate content problems caused by eCommerce faceted navigation (e.g. redundant URL combinations of colors, sizes, etc.). From what I've seen suggested methods include using AJAX or JavaScript to make the links functional for users only and prevent bots from crawling through them. I was wondering if this technique would work instead? If we detect that the user is a robot, instead of displaying a link, we simply display its anchor text. So what would be for a human COLOR < li > < a href = red >red < /a > < /li >
Intermediate & Advanced SEO | | anthematic
< li > < a href = blue>blue < /a > < /li > Would be for a robot COLOR < li > red < /li >
< li > blue < /li > Any reason I shouldn't do this? Thanks! *** edit Another reason to fix this is crawl budget since robots can waste their time going through every possible combination of facet. This is also something I'm looking to fix.0 -
CSS dropdown Navigation Structure for PR passing?
Hello, We are designing a very large site with hundreds of landing pages that will need to get some of the Pagerank and trust our homepage has, so we are trying to make sure our navigation architecure is well set up correctly from the beggining. I'm curious though if I need to have left side CSS dropdown navigation (I know no javascript) like www.adventurebound.com or if we can just use the top style dropdown like www.adventurefinder.com has? I know straight HTML links would be best but unfortunately our site will be too large and complex for that. Thanks in advance!
Intermediate & Advanced SEO | | iAnalyst.com0