How to prevent Google from crawling our product filter?
-
Hi All,
We have a crawler problem on one of our sites www.sneakerskoopjeonline.nl.
On this site, visitors can specify criteria to filter available products. These filters are passed as http/get arguments. The number of possible filter urls is virtually limitless.
In order to prevent duplicate content, or an insane amount of pages in the search indices, our software automatically adds noindex, nofollow and noarchive directives to these filter result pages. However, we’re unable to explain to crawlers (Google in particular) to ignore these urls.
We’ve already changed the on page filter html to javascript, hoping this would cause the crawler to ignore it. However, it seems that Googlebot executes the javascript and crawls the generated urls anyway.
What can we do to prevent Google from crawling all the filter options?
Thanks in advance for the help.
Kind regards,
Gerwin
-
The following is added to our robots.txt .. now lets wait and see the results
User-agent: * Disallow: /admin/
Disallow: /?
Allow /?product_date=&product_date2=*
Disallow /?product_date=&product_date2=&To check the working of the robots.txt i found a handy website;
-
The url looks like this;
http://www.sneakerskoopjeonline.nl/herensneakers?product_brand=
So just adding;
User-agent: *
Disallow: /*?product_brandShould do the trick?
Most important is that herensneakers itself should be indexed, followed and crawled -
I would use your robots.txt file to prevent them from crawling the specific strings / pages. Go into your Google Webmaster Tools and you can see all the information Google has on your site and any issues, you can also specify robots.txt information in there. That would be the best route as Google is obedient with what is on the robots.txt file. If you want more information about robots.txt, go here.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to rank a product page
Hello, I have a product page and just did a little introduction ( 5 lines of text) but I have the feeling this is not enough for google. What is the solution ? Is it to increase the amount of content even though it is not going to be user friendly ? Or will google look at the product page I link too and take into account the content on those subpages to boost my "pillar / product page". Thank you,
Intermediate & Advanced SEO | | seoanalytics1 -
Does google ignore ? in url?
Hi Guys, Have a site which ends ?v=6cc98ba2045f for all its URLs. Example: https://domain.com/products/cashmere/robes/?v=6cc98ba2045f Just wondering does Google ignore what is after the ?. Also any ideas what that is? Cheers.
Intermediate & Advanced SEO | | CarolynSC0 -
What exactly is an impression in Google Webmaster Tools search queries with the image filter turned on?
Is it when someone does an image search? Or does it count a regular search that has images in it? On an image search does the picture actually have to be viewed on the screen or can it be below in the infinite scroll?
Intermediate & Advanced SEO | | EcommerceSite0 -
Product Page on Eccomerce Site ranking very poorly - Unique Product description but duplicate content on other tabs.
Hi All, I have a query regarding my Product pages on my eCommerce site. I have unique Product descriptions but some of the other page content on the other tabs i.e Hire Terms , Delivery , About the Hire Company - Are duplicated across ALL my products. Is that okay or how should I deal with them ? See example url of one of my products below below - http://goo.gl/aSFPqP My products currently rank very badly... 200 + so Any advice would be greatly appreciated thanks Peter
Intermediate & Advanced SEO | | PeteC120 -
Missing Suite Number on Google
I realized that we are missing a suite number. It is not on the website or the recently updated Google/Bing/Yahoo revisions I did. Should I go and fix? Or should I go and adjust old listings. Does a suite number matter in the NAP?
Intermediate & Advanced SEO | | greenhornet770 -
Static links google guidelines
Google recommends to have static links it in guidelines Are breadcrumbs and static text link the same ? or in addition to breadcrumbs do I need static links on my page going from page A to B etc... The issue I have with static links this way is that if I look at the PR paper that would decrease the juice of my homepage ( which is the page I want to give the most juice to ) Thx,
Intermediate & Advanced SEO | | seoanalytics0 -
Google places
Is there away to get to the top of google places? Can it be manipulated?
Intermediate & Advanced SEO | | dynamic080