Layered navigation and hiding nav from user agent
-
I am trying to deal with the duplicate content issues presented by Magento's layered navigation feature (aka faceted navigation). I installed Amasty's Improved Navigation extension (https://amasty.com/improved-layered-navigation.html) and it offers the option to hide the layered navigation from specific user agents (ie googlebot, bingbot, etc).
This seems like cloaking to me and I hesitate to try it, unless hiding faceted navigation from specific user agents is known to be acceptable to Google (white hat practice). Does anyone know if this the case?
-
Great, thanks Carson! You're insights have been very helpful. I think we'll try to make the on-page ajax solution work.
-
If you're really worried about indexation I think that's a fine solution. It's definitely easier to manage, and it'll also be easier to track pageviews in most analytics platforms. The only downside is that if someone emails or links to a category page with filters applied the recipient won't see it. But generally people share products and not category pages, so it's not a big deal. I'd probably go that route.
Also make sure that your category pages still update the URL when you go to page 2, or that page 2 is somehow also being indexed. You don't want products that don't get indexed because categories can't be crawled.
-
Thanks for the link! I can see how Google offers me a way to tell it how to use my site variables. It seems like between managing parameters in webmaster tools, using canonical links and adding meta noindex tags on variable pages, there should be enough to keep things in order with the search engines. And I can just assume Google knows not to waste too much crawl budget on the variable pages.
I was considering one other option that would remove concerns about variables altogether. Using a different extension, I can set up Magento's layered navigation to work on the page without updating the URL. This eliminates the need for canonicals, parameters, and everything else that is more in Google's control than mine. What do you think of that as a solution?
-
Yes, the bots will crawl the pages, but they will not INDEX them.
There is a concern there, but mostly if the bots get caught in some kind of crawl trap - where they're trying out a near-infinite set of variables and getting stuck in a loop. Otherwise the spiders should understand the variables. You can actually check it in Webmaster tools to make sure Google understands. Instructions for that here:
https://support.google.com/webmasters/answer/6080550?hl=en
Ultimately Google will definitely not penalize you for having lots of duplicate content on URLs through variables, but it might be an issue with Googlebot not finding all your pages. You can make sure that doesn't happen by checking the indexation of your sitemap.
You could also try to block any URLs with the URL parameter in robots.txt. Make sure you get some help on the RegEx if you plan to do this. My advice is that blocking the variables in robots.txt is not worth it, as Google should have no problems with the variables - especially if the canonical tags are working.
Googlebot at least is smart enough these days to know when to stop crawling variable pages, so I think there are more important on-site things to worry about. Make sure your categories are linked to and optimized, for example.
-
This gets into an issue of bots and crawling where I am less clear. Even with canonicals, don't search engine bots crawl all of the pages produced with faceted navigation? That will easily reach 10,000+ pages on my site, which currently has a total number of pages in the low hundreds. I was under the impression I don't want to set up the faceted navigation in a way where the bots crawl through every combination of pages created by my products' attribute filters and bog the bots down in a quagmire of low-value pages. But I'm not sure if that's the case or how concerned I need to be about the bots spending their time on those pages.
-
If I'm not mistaken Magento has canonical tags on category pages by default, so you might be trying to solve an issue that doesn't exist. Take a look at the source code on faceted navigation to confirm. Or you can send me the site and I'll look over it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Googlebot being redirected but not users?
Hi, We seem to have a slightly odd issue. We noticed that a number of our location category pages were slipping off 1 page, and onto page 2 in our niche. On inspection, we noticed that our Arizona page had started ranking in place of a number of other location pages - Cali, Idaho, NJ etc. Weirdly, the pages they had replaced were no longer indexed, and would remain so, despite being fetched, tweeted etc. One test was to see when the dropped out pages had been last crawled, or at least cached. When conducting the 'cache:domain.com/category/location' on these pages, we were getting 301 redirected to, you guessed it, the Arizona page. Very odd. However, the dropped out pages were serving 200 OK when run through header checker tools, screaming frog etc. On the face of it, it would seem Googlebot is getting redirected when it is hitting a number of our key location pages, but users are not. Has anyone experienced anything like this? The theming of the pages are quite different in terms of content, meta etc. Thanks.
Intermediate & Advanced SEO | | Sayers0 -
H2 vs. H3 Tags for Category Navigation
Hey, all. I have client that uses tags in the navigation for its blog. For example, tags might appear around "Library," "Recent Posts," etc. This is handled through their WordPress theme. This seems fairly standard, but I wonder whether tags are semantically appropriate. Since each blog post is fairly lengthy (about 500-1000 words) with multiple tags, would it be more appropriate to use tags for this menu navigation? Are we cutting into the effectiveness of our tags by using them for menu navigation? The navigation is certainly an important page element, and it structures content, so it seems that it should use some header tag. Anyways, your thoughts are greatly appreciated. I'm a content creator, not an SEO, so this is a bit out of my skillset.
Intermediate & Advanced SEO | | Ask44435230 -
How should I manage duplicate content caused by a guided navigation for my e-commerce site?
I am working with a company which uses Endeca to power the guided navigation for our e-commerce site. I am concerned that the duplicate content generated by having the same products served under numerous refinement levels is damaging the sites ability to rank well, and was hoping the Moz community could help me understand how much of an impact this type of duplicate content could be having. I also would love to know if there are any best practices for how to manage this type of navigation. Should I nofollow all of the URLs which have more than 1 refinement used on a category, or should I allow the search engines to go deeper than that to preserve the long tail? Any help would be appreciated. Thank you.
Intermediate & Advanced SEO | | FireMountainGems0 -
How to Have Flat Navigation w/out Diluting Link Juice
I have a client with a very flat navigational structure relying on a menu with CSS hover dropdowns using simple items to get to just about every page on the site through the main navigation on every page. They want this ability to remain. The issue is stat is send link juice all over and does not concentrate into pages that are key search landing pages. I don't want to "no Index" "follow" the less important pages since there are some brand related long tail searches that I would want these pages found. These are useful pages to consumers who are already engaged with the brand, but not ones we would not care to rank for outside of branded search. If there was a way to have some links be non-crawled via javascript (or some other method) and those that are more important use a more standard html type link that would seem ideal. Does anyone have a suggestion for menu tool or technique for exposing to consumers all the links to consumers but restricting google bot's path while being in line with Google Webmaster guideslines? Blair
Intermediate & Advanced SEO | | BlairKuhnen0 -
Does hiding responsive design elements on smaller media types impact Google's mobile crawler?
I have a responsive site and we hide elements on smaller media types. For example, we have an extensive sitemap in the footer on desktop, but when you shrink the viewport to mobile we don't show the footer. Does this practice make Google's mobile bot crawler much less efficient and therefore impact our mobile search rankings?
Intermediate & Advanced SEO | | jcgoodrich1 -
Altering Breadcrumbs based on User Path to Product URL
Hi, Our products are listed in multiple categories, and as the URLs are path dependent (example.com/fruit/apples/granny-smith/, example.com/fruit/green-fruit/granny-smith/ and so forth) we canonicalise to the 'default' URL (in this case example.com/fruit/apples/granny-smith/). For mainly crawling bandwidth issues I'm looking to change all product URL's to path neutral so there is only ever one URL per product (example.com/granny-smith/), but still list the product in multiple categories. If a user comes directly to example.com/granny-smith/ then the breadcrumbs will use the default path "Fruit > Apples", however if the user navigated to the product via another category then I'd like the breadcrumbs to reflect this. I'm not worried about cloaking as it's not based on user-agent and it's very logical why it's being done so I don't expect a penalty. My question is - how do you recommend this is achieved from a technical standpoint? Many sites use path neutral product URL's (Ikea, PCWorld etc) but none alter the breadcrumbs depending upon path. Our site is mostly behind a CDN so it has to be a client side solution. I currently view the options as: Store Path to product in a cookie and/or browsers local-cache Attach the Path details after a # in the URL and use Javascript to alter breadcrumbs onload with JQuery When a user clicks to a product from a listing page, use AJAX to pull in the product info but leave the rest of the page (including the breadcrumbs) as-is, updating the URL accordingly Do you think any of these wouldn't work? Do you have a preference on which one is best? Is there another method you'd recommend? We also have "Next/Previous" functionality (links to the previous and next product URLs) on the page so I suspect we'd need to attach the path after a # and make another round trip to the server onload to update the previous and next links. Finally, does anyone know of any sites that do update the breadcrumbs depending upon path? Thanks in advance for your time FashionLux
Intermediate & Advanced SEO | | FashionLux1 -
Navigation
An e-commerce site I am working on currently displays 6 Super-Categories with a drop down that contains about 100 Categories for items which filter down to sub-cats and then the actual products. The issue is that every page starts off with these 100+ links just in navigation alone. I can only assume this is crippling our ability to spread link juice efficiently. I have looked at larger sites that have moved towards side navigation. A few examples: *amazon.com *walmart.com *newegg.com My issue is that we would like to move towards less links on the homepage to funnel our incoming links more efficiently but I cannot figure out how large sites cope with this. As far as I can tell they are using side nav that disappears after selecting a category of item in which the navigation is replaced with filtering tools and the nav is hidden above (see the sites above). Is this the best way to handle this issue? Also is there a way to find out exactly what they are doing because I am trying to explain this to our IT person and I just get a response that our site is fine how it is and these navigation links don't affect anything...even though each page starts off with the same 100 follow links of navigation. Thanks
Intermediate & Advanced SEO | | MichealGooden0 -
Navigation - Balancing UX & SEO
I'm currently evaluating our navigation in the course of a site relaunch. From reading a number of articles and posts on seoMOZ, here are the elements I've found important to consider: Use CSS (not Javascript) for the primary drop-down navigation menu Get rid of two design elements from our earlier days: The 30 something site-wide category links in the footer, and many no-followed internal links (in an attempt to sculpt PR) Keep all pages within 3 clicks of the homepage, and have ample cross-links within internal pages. The one major problem I'm facing is how to balance UX and SEO in the primary navigation bar. To illustrate, let's assume I sell Tennis equipment. If one of the top-level categories on my navigation bar was "Rackets", if I was designing purely with SEO in mind the category names would be: Tennis Rackets -> Wilson Tennis Rackets Head Tennis Rackets Prince Tennis Rackets ....as the full, three word anchor text will be most specific and valuable to pass reputation to the category pages. However, from a UX perspective, writing "Tennis Rackets" after each category is unnecessary, and it would look MUCH cleaner to instead have: Tennis Rackets -> Wilson Head Prince ....but this would obviously be less beneficial from a SEO standpoint for each individual, manufacturer racquet page as the entire search term ("Wilson Tennis Rackets") is not in the anchor text. As these links will be on every page of the site, I'm struggling with which to choose - clean navigation or improved SEO. My Questions: I would love to hear the communities thoughts on how to weigh the balance of these two - clean UX navigation vs. SEO-rich specific anchor text - in navigation. Also, I'd appreciate hearing if any of my original 3 assumptions for the re-design are off-base or incorrect. Thank you!
Intermediate & Advanced SEO | | AndrewY0