Search directory - How to apply robots
-
Hi.
On the site I'm working on, we use a search directory to display our search results. It displays as follows -
With the dynamic search results appearing after the hash tag.
Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav.
How should I handle this?
Thanks.
-
That was my first instinct as well - but I wanted to check if there was a better move? Thanks for your help!
-
Have you considered switching to simply "noindex, follow" for these pages? This would both keep them out of the search index and would allow for the benefit of external links to these pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Reason for robots.txt file blocking products on category pages?
Hi I have a website with thosands of products. On the category pages, all the products are linked to with the code “?cgid” in the URL. But “?cgid” is also blocked in the robots.txt file for some reason. So I'm thinking it's stopping all my products getting crawled by Google. Am I right here? Is there any reason why a website would want to limit so many URL's? I'm only here a week and the sites getting great traffic, so don't want to go breaking it!!! Thanks
Web Design | | Frankie-BTDublin0 -
Map Search Tools to integrate on Accommodation Website
Hello all, Can any recommend a Map Search tool that i can integrate into my accommodation website. Ideally I want to be able to pin my clients on the map search with links back to their listing page on my website and provide an alternative search facility for clients looking for accommodation. Am covering the South Africa region specifically. Am assuming that i could go down the Google Maps route but would really like to know what alternatives there are on offer. Also what SEO considerations do i need to think about when adding this to my website. Thanks in advance for any help.
Web Design | | SamanthaRiggien0 -
Empirical Data on the effect of embedded Google Maps on Search Ranking
Does anyone have any empiric data on the effect of an embedded map on SERP's? Please understand that I already have anecdotal info and a personal opinion. I am looking for data. Thanks
Web Design | | RobertFisher0 -
How do search engines interpret <hgroup>...</hgroup> tags?
Hi there. I'm building an HTML 5 site and through research of new HTML 5 elements I've seen little conclusive information about the interpretation of the new <hgroup>tag, in terms of SEO application and interpretation. In particular does Google interpret the nested heading tags as individual elements or does it combine them into one entity? For example, say I have: <hgroup> # Article Heading ## Article Sub-heading </hgroup> How is this interpreted by Google and what would be some good SEO best practices regarding the <hgroup>tag in HTML5: Is it interpretted as a single tag (" Article Heading: Article Sub-heading ") or two separate heading tags (one and one )? Also, how much does the ordering of the tags matter (say for example I wanted something like the following for visual purposes? <hgroup> ## Article Sub-heading # Article Heading </hgroup> One last thing: is it safe to assume that it is indeed OK to have multiple tags on a single page (as referenced by Matt Cutts a while back in a Webmaster Video)? Thanks! </hgroup> </hgroup>
Web Design | | LMDNYC2 -
Page Title or Search Friendly Urls?
We are currently auditing our website as part of our SEO strategy. One item which hascome up is the importance of search friendly urls against the search engine friendly page titles. Do url's or page titles carry more relevance than the other in search engines? Obviously the ideal would be to have both to maximise search impact but do either carry more importance. Thanks
Web Design | | bwfc770 -
The primary search keywords for our news release network have dropped like a rock in Google... we are not sure why.
Hi, On April 11th, a month after the farmer update was released for U.S. users of Google, the primary keywords for ALL our sites significantly dropped in Google. I have some ideas why, but I wanted to get some second opinions also. First off, I did some research if Google did anything on the 11th of April... they did. They implemented the farmer update internationally, but that does not explain why our ranks did not drop in March for U.S. Google users... unless they rolled out their update based on what site the domain is registered in... in our case, Canada. The primary news release site is www.hotelnewsresource.com, but we have many running on the same server. EG. www.restaurantnewsresource.com, www.travelindustrywire.com and many more. We were number 1 or had top ranks for terms like ¨Hotel News¨, ¨Hotel Industry¨, ¨Hotel Financing¨, ¨Hotel Jobs¨, ¨Hotels for Sale¨, etc... and now, for most of these we have dropped in a big way. It seems that Google has issued a penalty for every internal page we link to. Couple obvious issues with the current template we use... too many links, and we intend to change that asap, but it has never been a problem before. The domain hotelnewsresource.com is 10 years old and still holds a page rank of 6. Secondly, the way our news system works, it´s possible to access an article from any domain in the network. E.G. I can read an article that was assigned to www.hotelnewsresource.com on www.restaurantnewsresource.com... we don´t post links to the irrelevant domain, but it does sometimes get indexed. So, we are going to implement the Google source meta tag option. The bottom line is that I think we put too much faith in the maturity of the domain... thinking that may protect us... not the case and it´s now a big mess. Any insight you can offer would be greatly appreciated. Do you think it was farmer or possibly something else? Thanks, Jarrett
Web Design | | jarrett.mackay0 -
Correct use for Robots.txt
I'm in the process of building a website and am experimenting with some new pages. I don't want search engines to begin crawling the site yet. I would like to add the Robot.txt on my pages that I don't want them to crawl. If I do this, can I remove it later and get them to crawl those pages?
Web Design | | EricVallee340