Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
How to exclude URL filter searches in robots.txt
-
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505
How can I exclude all of these filters in the robots.txt? I think it'll be:
Disallow: /*?color=$
Is that the correct syntax with the $ sign in it? Thanks!
-
Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.
-
Yes, but what about bing and rest of Search Engine?
-
Adrian,
I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.
It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.
And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.
- On the Webmaster Tools Home page, click the site you want.
- Under Crawl, click Blocked URLs.
- If it's not already selected, click the Test robots.txt tab.
- Copy the content of your robots.txt file, and paste it into the first box.
- In the URLs box, list the site to test against.
- In the User-agents list, select the user-agents you want (e.g. Googlebot)
-
There certainly is a right answer to my question - I already posted it here earlier today:
Disallow: /*?color=
Disallow: /?*manufacturer=Without the $ at the end which would otherwise denote the end of the URL.
-
Hello Adrian,
The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.
Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:
(Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar
)Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate ContentI don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...
Good luck!
-
It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)
-
I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:
To exclude pages with color or manufacture in them you can use
Disallow: /*?color=
Disallow: /?*manufacturer=A question mark in your try should be omitted as it denotes the end of the url
-
Hi
I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.
Hope this helps.
Steve
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are the prepositions and separate letters in URL bad for website optimization?
Is it ok for website optimization to use prepositions and separate letters in URL ? Examples: -i-series ; -salad-with-avocado etc.
On-Page Optimization | | adrecom0 -
Will changing a URL negatively affect ranking?
Hello Mozzers, We would like to change the URL for a page on our website which ranks well for some our keyphrases/words. We are hoping the change of URL, through the addition of an additional keyword would help boost the rank of that URL further. At the moment out page gets 2 x A and 2 x B 1xF on the MOZ page rank tool using 5 keyphrase/word variations . One phrase ranks 4, one ranks 3 and the other 3 are 'not in the top 50' Our plan was to change the URL, using SHF404, and use 'Fetch' in the Google search console to re-submit the page to Google. Appreciate you can't give any guarantees how Google will behave, just wondered what your thoughts were on the wisdom of changing the URL in the first place? Thanks Ian
On-Page Optimization | | Substance-create0 -
Should I use an acronym in my URL?
I know that Google understands various acronyms. Example: If I search for CRM System, it knows i'm searching for a customer relationship management system. However, will it recognize less known acronyms? I have a page geared specifically for SAP data archiving for human capital management systems. For those in the industry, they simply call it HCM. Here is how I view my options: Option #1: www.mywebsite.com/sap-data-archiving/human-capital-management Option #2: www.mywebsite.com/sap-data-archiving/hcm Option #3: www.mywebsite.com/sap-data-archiving/hcm-human-capital-management With option #3, i'm capturing the acronym AND the full phrase. This doesn't make my URL overly long either. Of course, in my content i'll reference both. What does everyone else think about the URL? -Alex
On-Page Optimization | | MeasureEverything0 -
How important are clean URLs?
Just wanting to understand the importance of clean URLs in regards to SEO effectiveness. Currently, we have URLs for a site that reads as follows: http://www.interhampers.com.au/c/90/Corporate Gift Hampers Should we look into modifying this so that the URL does not have % or figures?
On-Page Optimization | | Gavo1 -
ECommerce Filtering Affect on SEO
I'm building an eCommerce website which has an advanced filter on the left hand side of the category pages. It allows users to tick boxes for colours, sizes, materials, and so on. When they've made their choices they submit (this will likely be an AJAX thing in a future release, but isn't at time of writing). The new filtered page has a new URL, which is made up of the IDs of the filter's they've ticked - it's a bit like /department/2/17-7-4/10/ My concern is that the filtered pages are, on the most part, going to be the same as the parent. Which may lead to duplicate content. My other concern is that these two URLs would lead to the exact same page (although the system would never generate the 'wrong' URL) /department/2/17-7-4/10/ /department/2/**10/**17-7-4/ But I can't think of a way of canonicalising that automatically. Tricky. So the meat of the question is this: should I worry about this causing issues with the SEO - or can I have trust in Google to work it out?
On-Page Optimization | | AndieF0 -
Question about url structure for large real estate website
I've been running a large real estate rental website for the past few years and on May 8, 2013 my Google traffic dropped by about 50%. I'm concerned that my current url structure might be causing thin content pages for certain rental type + location searches. My current directory structure is:
On-Page Optimization | | Amped
domain.com/home-rentals/california/
domain.com/home-rentals/california/beverly-hills/
domain.com/home-rentals/california/beverly-hills/90210/
domain.com/apartment-rentals/california/
domain.com/apartment-rentals/california/beverly-hills/
domain.com/apartment-rentals/california/beverly-hills/90210/
etc.. I was thinking of changing it to the following:
domain.com/rentals/california/
domain.com/rentals/california/beverly-hills/
domain.com/rentals/california/beverly-hills/90210/ ** Note: I'd provide users the ability to filter their results by rental type - by default all types would be displayed. Another question - my listing pages are currently displayed as:
domain.com/123456 And I've been thinking of changing it to:
domain.com/123456-123-Street-City-State-Zip Should I proceed with both changes - one or the one - neither - or something else I'm not thinking of? Thank you in advance!!0 -
Does 'XXX' in Domain get filtered by Google
I have a friend that has xxx in there domain and they are a religious based sex/porn addiction company but they don't show up for the queries that they are optimized against. They have a 12+ year old domain, all good health signs in quality links and press from trusted companies. Google sends them adult traffic, mostly 'trolls' and not the users they are looking for. Has anyone experienced domain word filtering and have a work around or solution? I posted in the Google Webmaster help forums and that community seems a little 'high on their horses' and are trying to hard to be cool. I am not too religious and don't necessarily support the views of the website but just trying to help a friend of a friend with a topic that I have never encountered. here is the url: xxxchurch.com Thanks, Brian
On-Page Optimization | | Add3.com0 -
Incoming Search Terms
Hi guys, I saw a blog post recently where the author added a list of "incoming search queries" to the bottom of his post, obviously to improve the post's ranking for those terms. On one hand, I suppose it it does help users find that post. On the other, it seems lazy and somewhat dodgy, but I haven't found any opinions on it elsewhere and have not seen this practice in my experience. What're your thoughts? Outright search engine manipulation? Cheers, Carlo SCWYt
On-Page Optimization | | mtgconsulting0