How to exclude URL filter searches in robots.txt
-
When I look through my MOZ reports I can see it's included 'pages' which it shouldn't have included i.e. adding filtering rules such as this one http://www.mydomain.com/brands?color=364&manufacturer=505
How can I exclude all of these filters in the robots.txt? I think it'll be:
Disallow: /*?color=$
Is that the correct syntax with the $ sign in it? Thanks!
-
Unless you're specifically calling out Bing or Baidu... in your Robots.txt file they should follow the same directives as Google so testing with Google's Robots.txt file tester should suffice for all of them.
-
Yes, but what about bing and rest of Search Engine?
-
Adrian,
I agree that there certainly is a right answer to the question posted, as the question asks specifically about one way to manage the issue, being a block of filters in the robots.txt file. What I was getting at is that this may or may not necessarily be the "best" way, and that I'd need to look at your site and your unique situation to figure our which would be the best solution for your needs.
It is very likely that with these parameters a robots.txt file block is the best approach, assuming the parameters aren't added by default into category page or category pagination page navigational links, as then it would affect the bot's ability to crawl the site. Also, if people are linking to those URLs (highly unlikely though) you may consider a robots meta noindex,follow tag instead so the pagerank could flow to other pages.
And I'm not entirely sure the code you provided above will work if the blocked parameter is the first one in the string (e.g. domain.com/category/?color=red) as there is the additional wildcard between the ? and the parameter. I would advise testing this in Google Webmaster Tools first.
- On the Webmaster Tools Home page, click the site you want.
- Under Crawl, click Blocked URLs.
- If it's not already selected, click the Test robots.txt tab.
- Copy the content of your robots.txt file, and paste it into the first box.
- In the URLs box, list the site to test against.
- In the User-agents list, select the user-agents you want (e.g. Googlebot)
-
There certainly is a right answer to my question - I already posted it here earlier today:
Disallow: /*?color=
Disallow: /?*manufacturer=Without the $ at the end which would otherwise denote the end of the URL.
-
Hello Adrian,
The Moz reports are meant to help you uncover issues like this. If you're seeing non-canonical URLs in the Moz report then there is a potential issue for Google, Bing and other search engines as well.
Google does respect wildcards (*) in the robots.txt file, though it can easily be done wrong. There is not right or wrong answer to the issue of using filters or faceted navigation, as each circumstance is going to be different. However, I hope some of these articles will help you identify the best approach for your needs:
(Note: Faceted Navigation is not exactly the same as category filters, but the issues and possible solutions are very similar
)Building Faceted Navigation That Doesn't Suck Faceted Navigation Whiteboard Friday
Duplicate Content: Block, Redirect or Canonical
Guide to eCommerce Facets, Filters and Categories
Rel Canonical How To and Why Not
Moz.com Guide to Duplicate ContentI don't know how your store handles these (e.g. does it add the filter automatically, or only when a user selects a filter?) so I can't give you the answer, but I promise if you read those articles above you will have a very good understanding of all of the options so you can choose which is best for you. That might end up being as simple as blocking the filters in your robots.txt file, or you may opt for rel canonical, noindex meta tag, ajax, Google parameter handling, etc...
Good luck!
-
It's not Google's index that I'm interested in in this case, it's for the MOZ reports. Moz was including over 10,000 'pages' because it was indexing these URLs. Now I know how to edit the robots.txt Moz will be prevented from indexing them again (we only have around 2,000 real pages, not 10,000)
-
I sought out the answer from a developer and got the following reply, so posting here in case it helps someone else:
To exclude pages with color or manufacture in them you can use
Disallow: /*?color=
Disallow: /?*manufacturer=A question mark in your try should be omitted as it denotes the end of the url
-
Hi
I would recommend excluding these in Google Webmaster Tools. Once logged in to your account under the "Crawl" menu you will find "URL Parameters". Find the relevant parameter in the list on this page and you can tell Google not to index these pages.
Hope this helps.
Steve
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL Keywords
I am doing SEO on our eCommerce website and read that I should include keywords in the URL The original URL is: http://thegiftlinks.com/personalized-wedding-glass.html
On-Page Optimization | | abdulw
Title page: Wedding gift Dubai - Anniversary gift Dubai - Personalized Wedding Glass
Meta Data:
Wedding gift Dubai - Anniversary gift Dubai - Personalized Wedding Glass
It is great for a wedding gift and anniversary gift for friends and family members. If I will include the keyword to the url it will be like this
http://thegiftlinks.com/personalized-wedding-glass.html/Wedding-gift-Dubai is this the correct way to include keywords in the URL? Thanks0 -
URL Path. What is better for SEO
Hello Moz people, Is it better for SEO to have a URL path like this: flowersite.com/anniversary_flowers/dozen_roses OR flowersite.com/dozen_roses Is it better to have the full trail of pages in the URL?
On-Page Optimization | | CKerr0 -
URL structure
Hello all, I am about to sort out my websites link structure, and was wondering which approach to our services page would be best. should we have: services/digital-marketing & services/website-design etc or: digital-marketing/website-design & digital-marketing/seo Basically I see digital marketing as the top level category that is the umbrella term for all of our digital services. But would it make more sense to have service to be the main category and digital marketing within that (along with all the other services from web design to seo)? all thoughts welcome!
On-Page Optimization | | wseabrook0 -
Do parameters in a URL make a difference from an SEO point of view
We us a number of different parameters in a number of our URLs to track how the user has navigated to the page. So for example we will have a page www.example.com/product/?banner to show that the user has navigated to the page from the banner as opposed to www.example.com/product/?footer to show that the user has navigated to the page from the footer. Do search engines treat these pages as the same page or different pages? Thanks
On-Page Optimization | | cbarron0 -
Best URL structure for my page
Hey everyone, I am wondering what is the best URL structure for my activity booking page in Bali. http://www.thingstodoinbali.com/bali/sports/water-sports/rafting/ or http://www.thingstodoinbali.com/bali/sports/water-sports/rafting-bali/ or would you recommend something else? Most people would google for "activity bali" in this case "rafting bali". Please advice. Thanks for your time and help in advance!
On-Page Optimization | | c.eiermann0 -
How to Structure URL's for Multiple Locations
We are currently undergoing a site redesign and are trying to figure out the best way to structure the URL's and breadcrumbs for our many locations. We currently have 60 locations nationwide and our URL structure is as follows: www.mydomain.com/locations/{location} Where {location} is the specific street the location is on or the neighborhood the location is in. (i.e. www.mydomain.com/locations/waterford-lakes) The issue is, {location} is usually too specific and is not a broad enough keyword. The location "Waterford-Lakes" is in Orlando and "Orlando" is the important keyword, not " Waterford Lakes". To address this, we want to introduce state and city pages. Each state and city page would link to each location within that state or city (i.e. an Orlando page with links to "Waterford Lakes", "Lake Nona", "South Orlando", etc.). The question is how to structure this. Option 1 Use the our existing URL and breadcrumb structure (www.mydomain.com/locations/{location}) and add state and city pages outside the URL path: www.mydomain.com/{area} www.mydomain.com/{state} Option 2 Build the city and state pages into the URL and breadcrumb path: www.mydomain.com/locations/{state}/{area}/{location} (i.e www.mydomain.com/locations/fl/orlando/waterford-lakes) Any insight is much appreciated. Thanks!
On-Page Optimization | | uBreakiFix0 -
Maximum length of a URL for good SEO?
Hi there, We have a content database as part of our site and I noticed that the way the database is loaded with new content, it means that the URL’s for these pages are really long, around 100 characters or sometimes more http://www.xxxyy.org/knowledge-base/documents/word1-word2-word3-word4-word5-word6-word7-word8 Is there a suggted maximum character length for a URL? Kind of like for title tag where I max out at 69… Should I truncate the URL’s or at least reduce the numbers of words in them to something less spammy? Does that make a difference? Thanks!
On-Page Optimization | | inhouseninja0 -
URL length does it really make a difference
I am currently working redevloping my site, I have terribly long URL's according to SEOMoz, can someone cover the real benefit of my editing my URLs, and then putting in all of those redirects. Should I edit old material? My site has been in wordpressformat for a couple of years now.
On-Page Optimization | | copykatrecipes0