How to safely exclude search result pages from Google's index?
-
Hello everyone,
I'm wondering what's the best way to prevent/block search result pages from being indexed by Google. The way search works on my site is that search form generates URLs like:
/index.php?blah-blah-search-results-blahI wanted to block everything of that sort, but how do I do it without blocking /index.php ?
Thanks in advance and have a great day everyone!
-
Hi Louise,
If you can ID the parameters, you can also look at blocking these in Webmaster Tools. This page explains more. As with any blocking of URLs, of course, proceed with caution.
-
I agree that can be effective. The reason I suggested the robots.txt is because Louise mentioned "blocking and preventing" as an objective. Robots.txt are particularly useful in the example where results from a search bar or something of that nature is involved. A NOINDEX, FOLLOW will not prevent bots from getting tired and dizzy, whereas the robots.txt can "block and prevent" bots from crawling certain parameters.
With all of that said, I think it is important to understand whether you need the bots to crawl and not index (in which case Spencer's answer is correct), or if you need to prevent bots from crawling the parameters altogether.
Hope that is more clear
-
I'm not sure that robots.txt is effective when url parameters are involved.
I would just add a meta robots tag to the head section of the search results template:
-
If you are able to identify a url parameter, you may excluded them using robots.txt. Here is a great resource on Robots.txt - http://moz.com/learn/seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Shall i index double pages of my website as compared to my competitors?
a:my competitors has indexed 10 pages (checked it with site:abcd.com and found 10 results) b:what if i index 20 pages of my site and create a lot of content which is also better than my competitors who will have the edge?
Reporting & Analytics | | calvinkj0 -
More 'New users' than 'Users' in Google Analytics
Hey Mozzers, I'm seeing more 'New users' than 'Users' in Google Analytics (see screenshot). This is over a 3 year time period. Any idea why new users would be greater than users? Thanks! 6lYmU0j
Reporting & Analytics | | RWesley0 -
Page Views for a List of URL's (Excel & Analytics)
I've been playing around with a few excel plugins trying to get the visit count for over 350 different URL's on the same website. I cant seem to find a function that can do this. Can anyone advise a work around or a way i can achieve this? Greg
Reporting & Analytics | | AndreVanKets0 -
Using Filters On Google Analytics
Hi Guys, I am In need of some technical help with regards to Google analytics. One of my clients has a site providing HR Software on SaaS basis. The users login via the site for the various HR Tools. Obviously this is skewing the analytics data and showing 10,000+ extra visitors that actually don't exist. Now I understand i can add filters and exclude certain pages for instance the login page, however, some users will visit the home page then click login. How can I exclude them just to show new visitors to the site rather than users that are just trying to login. Hope this makes sense Neil
Reporting & Analytics | | nezona0 -
Setting up goals in Google Analytics
Hello! This question is seems so obvious that I'm also ashamed to ask... almost. If the goal of a website is to have a visitor complete a contact form, when setting up the Goal in Google Analytics (as a "URL destination") the URL of choice would be the form's thank you page, correct? Because that's the page that proves the visitor completed the task we want them the achieve. Right? Thanks!
Reporting & Analytics | | SmileMoreSEO
Erik0 -
Mobile Site on Google Analytics
Hi mozzers, We just launched a mobile site and I was wondering what are the main steps to follow for gettting your mobile site tracked via GA (m.example.com)? We have a profile for www.example.com GATC: javascript or PHP to install? Should the profile be on a subdomain? What else to consider when implementing a mobile site on GA? Thanks
Reporting & Analytics | | Ideas-Money-Art0 -
How can I track search engine optimization data in Google analytics?
My website is linked to a Google Analytics web property. But, I am not able to track search engine optimization data in Google Analytics. So, How can I get it done?
Reporting & Analytics | | CommercePundit0 -
Google vs Bing/Yahoo SERP results?
I see large differences in Google and Bing/Yahoo SERP results for many keywords. Google shows many of our primary keywords in their top ten, while Bing/Yahoo rank the same one 40-50 or above. Do you have any insight on their differences that might account for this variation? We are legitimate, long time, white hatters at a small manufacturer that is one of the leaders in our industry. The only thing I can think of that might describe this issue is PPC advertising. Their algorithms wouldn't be that inconsistent would they? (We do much more paid advertising on Google) But generally this should have no affect, right? Thanks, George...
Reporting & Analytics | | rhawk0