How to exclude URL filter searches in robots.txt
-
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505
How can I exclude all of these filters in the robots.txt? I think it'll be:
Disallow: /*?color=$
Is that the correct syntax with the $ sign in it? Thanks!
-
Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.
-
Yes, but what about bing and rest of Search Engine?
-
Adrian,
I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.
It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.
And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.
- On the Webmaster Tools Home page, click the site you want.
- Under Crawl, click Blocked URLs.
- If it's not already selected, click the Test robots.txt tab.
- Copy the content of your robots.txt file, and paste it into the first box.
- In the URLs box, list the site to test against.
- In the User-agents list, select the user-agents you want (e.g. Googlebot)
-
There certainly is a right answer to my question - I already posted it here earlier today:
Disallow: /*?color=
Disallow: /?*manufacturer=Without the $ at the end which would otherwise denote the end of the URL.
-
Hello Adrian,
The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.
Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:
(Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar
)Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate ContentI don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...
Good luck!
-
It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)
-
I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:
To exclude pages with color or manufacture in them you can use
Disallow: /*?color=
Disallow: /?*manufacturer=A question mark in your try should be omitted as it denotes the end of the url
-
Hi
I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.
Hope this helps.
Steve
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Need advice on the better URL structure to go with
I am rebuilding our existing website on a new platform and need advice on which URL structure would be the most ideal. The following examples are of a product that we have with a very long page title. Not all of our products have titles this long, but enough of them do to cause some concern. I was also wondering if I should end the url with file type .html or if leaving it out is better. Thanks in advance! OPTION 1. this example just uses the root domain and the entire product title separated by dashes http://ewheels.nextmp.net/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 2. this example uses the crawl path as well as the entire product title http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/staggered-full-set-br-2-20x9-ace-alloy-aff01-metallic-silver-machined-face-flow-formed-br-2-20x10-5-ace-alloy-aff01-metallic-silver-machined-face-flow-formed OPTION 3. this example uses the crawl path and just the part number at the end since the folders already contain all the keywords necessary http://ewheels.nextmp.net/wheels/ace-alloy-wheels/ace-alloy-aff01-metallic-silver-machined-face-flow-formed/ace-2090aff01silace-20105aff01sil
On-Page Optimization | | elementmotor0 -
Google search returns blog homepage, but not article
When I do a google search for a specific article on our blog the search results only return the blog homepage with the article title shown in the meta description, but never the actual article page. I've tried to refine my search by using site: and quotation marks around the article title (e.g site:www.example.com "article title") but still only get the homepage. Our blog is showing up so I assume it's not an indexing issue, but not sure how to get the article pages to show on serps. Any ideas? Thanks!
On-Page Optimization | | STP_SEO0 -
What word should I use in my URL for my blog
Should I use the word "blog" in my sub folder as in : http://www.mybusiness.com/blog or should I use http://www.mybusiness.com/news. Is there a difference for when my site is crawled. I understand that a blog works a little differently. Can someone explain the basics?
On-Page Optimization | | graemesanderson0 -
Google Search - One page having problems
this issue is concerning my site - cruvoir.com we retail designer clothing online, and currently have 17 'designer' pages - one for each manufacturer brand name. We target these brand names for our campaign and track the progress with Moz and try to focus them in Google search. Of many of the designer names, we rank pretty well in Google search (usually under #15 when searching for the specific brand. All brands are doing well, except one brand : "Lost And Found" - a designer label we carry. This is the page for this brand name : https://cruvoir.com/5-lost-and-found we cannot figure it out. It happens to be our most important label we carry. when we search for this brand name or include it in any other search terms, we never are in the google search results. I expect it is a crawl issue, but we have covered all our ground in optimizing this brand page. It seems this page is also indexed with Google. But we cannot figure out why it does not rank us in search.
On-Page Optimization | | cruvoir0 -
USA appears in search snippet
Buongiorno from 14 degrees C very cloudy Wetherby UK, When I type "Dartex Coatings" in Google i get a search snippet with USA in the description. See illustration: http://i216.photobucket.com/albums/cc53/zymurgy_bucket/usa-indescription-2_zps42296e0a.jpg I cannot see where this is comming from on site http://www.dartexcoatings.com/ . HAving checked the souce code there is no reference to USA 😞 So my question is please..."where is Google picking up USA from"? Grazie,
On-Page Optimization | | Nightwing
David0 -
Author avatar in homepage search result
Hello I actually see some websites which the owner don't want to display the google+ profile avatar when homepage is displayed in search result, except blog posts only For example: yoast.com Is it a good practice in wordpress? or we must claim authorship & display the google+ avatar for all pages & all posts? Thanks
On-Page Optimization | | dklong23450 -
Shall Google index a search result?
Hi, I've a website with about 1000 articles.Each article has one ore more keywords / tags. So I display these keywords at the article page and put a link to the intern search engine. (Like a tag cloud) The search engine lists als articles with the same keyword and creates a result page. This result page is indexed by Google. The search result contains the title of the article, a short description (150-300 chars.) and a link to the article. So, Google believes, that there are about 5.000 pages instead of 1.000 because auf the link to the search result pages. The old rule was for me: More pages in Google = better. But is this still true nowadays? Would be a "noindex, follow" better on these search result pages? (Is there a way to tell Google that this is a search result page?) Best wishes, Georg.
On-Page Optimization | | GeorgFranz0