How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Webmaster Tools Pages and Queries
In Google Webmaster Tools, under Search Analytics, what is the difference between positions in queries and positions in pages?
Reporting & Analytics | | CostumeD0 -
On Google Analytics, Pages that were 301 redirected are still being crawled. What's the issue here?
URL that we redirected are being crawled on Google Analytics. Since they dont exist, they have high bounce rates. What can the issue be?
Reporting & Analytics | | prestigeluxuryrentals.com0 -
Is Google doing anything about the spam referrer plague in Google Analytics?
My client manages 15+ websites and we have observed the spam referrer issue has gotten steadily worse and worse in the last few months. I took the time to weed out the worst offenders via traffic filters but our referral traffic source garden was quickly full of weeds again a few weeks later. Does anybody know what the Google Analytics development team is doing to combat the spam referrer issue? I find it quite amazing that Google goes through the asinine effort to block out organic keyword data ("not found" & "not provided) but they don't seem to care about blacklisting/filtering spam referrers. RHnE6Wx
Reporting & Analytics | | RosemaryB3 -
How do I fix 608's please?
Hi, I'm on the free trial and finding it very useful I've fixed all my 301's. but now I have a load of 608's. I don't no what this is! I feel like I've cured herpes only to get gonorrhea! can any one help. I have 41 608's which is more than the 301's I had. I hope they are non-related! I won't bore you with the whole list but some of the url's are: Error Code 608: Page not Decodable as Specified Content Encoding http://sussexchef.com/catering-at-mr-mrs-currys-50th-wedding-anniversary/guestsarrive----608 Error Code 608: Page not Decodable as Specified Content Encoding http://sussexchef.com/funeral-catering/picture4-2----608 Error Code 608: Page not Decodable as Specified Content Encoding http://sussexchef.com/wedding-venues
Reporting & Analytics | | SussexChef831 -
How come de results from the old Google Keyword Tool are so different from the results from the keyword planner?
When I insert the keyword 'IT jobs' with Belgium as country and Dutch as the language, there is a huge difference between the results in the keyword planner and the keyword tool. Keyword tool says 1,000,000 local searches per month Keyword planner says 590 local searches per month This is a really big difference and I don't know which results I should trust. Can anybody help me with this?
Reporting & Analytics | | Murielleu0 -
Why are plus signs (+) suddenly showing up in Google Analytics organic search keywords reports?
Since June 13, 2013, the number of organic search queries containing a plus sign (+) has gone up over 1,000% compared to the previous period on my site in Google Analytics. These plus signs appear to be taking the place of spaces in these search queries (i.e. "word1+word2+word3"). This appears to be almost (or completely) Google organic traffic, not other search engines. Since I highly doubt searcher behavior would change so suddenly, I'm trying to figure out why Google is replacing spaces with plus signs. Is anyone else seeing this? Any ideas?
Reporting & Analytics | | RCF0 -
Get a list of robots.txt blocked URL and tell Google to crawl and index it.
Some of my key pages got blocked by robots.txt file and I have made required changes in robots.txt file but how can I get the blocked URL's list. My webmaster page Health>blocked URL's shows only number not the blocked URL's.My first question is from where can I fetch these blocked URL's and how can I get them back in searches, One other interesting point I see is that blocked pages are still showing up in searches.Title is appearing fine but Description shows blocked by robots.txt file. I need urgent recommendation as I do not want to see drop in my traffic any more.
Reporting & Analytics | | csfarnsworth0 -
Google Analytice for Shopping Results
We are trying to optimise for Google Shopping and therefore would like to be able to monitor traffic to site from this source? How can this be done in Google Analytics? Thanks
Reporting & Analytics | | seanmccauley0