Search directory - How to apply robots
-
Hi.
On the site I'm working on, we use a search directory to display our search results. It displays as follows -
With the dynamic search results appearing after the hash tag.
Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav.
How should I handle this?
Thanks.
-
That was my first instinct as well - but I wanted to check if there was a better move? Thanks for your help!
-
Have you considered switching to simply "noindex, follow" for these pages? This would both keep them out of the search index and would allow for the benefit of external links to these pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console Block
Am new to SEO. My clients site was completed using Yoast premium and then used Google search console to initiate the crawl. Initially setup an http:// property and all seemed good. Then i removed that under search console an created an https:// did the render and it appears google has put a block and placed their own robots.txt file which basically has rendered the site useless. Feedback most appreciated.
Web Design | | BoostMyBiz0 -
How to command Robots.txt to this:
Hi, So for some reason I have this unexplained issues in webmaster tools. Check them out: http://prntscr.com/7n1nj8 See that iSeeCars.com? How to remove it? Is it just disallow: iseecars.com? Or should I disallow the search to be crawled? Regards,
Web Design | | Kokolo0 -
Creating Multiple Sub-Directories in Wordpress
Hi fellow Mozzers, I'm currently in the process of planning/building a website for e-commerce and have stumbled on a bit of a hurdle with sub-directories. I want to use a piece of software called SellerDeck to generate my e-commerce store and also my homepage (index.html). This element of the build is fine as the e-commerce store will sit in a sub-directory of .co.uk/store/. What I'm struggling with is the rest of the site architecture. I want to use Wordpress to manage content for the rest of the site. I want to have sub-directories .co.uk/help/ and .co.uk/blog/, all managed from one Wordpress installation. Is this possible? If not, does having two separate installations of Wordpress create any speed issues? Additional question for bonus points from me; lets say I wanted to do away with sub-directories for the /help/ and /blog/ elements (but keep the /store/), could I have a Wordpress installation that doesn't generate a homepage (index.php) so I can utilise the e-commerce software version instead. Essentially I'd be installing Wordpress at the root folder, but wouldn't have an index.php made by Wordpress. Many thanks in advance
Web Design | | BlueTree_Sean0 -
The impact of using directories without target keyword on our Rankings
Hello all, I have a question regarding a website I am working on. I’ve read a lot of Q en A’s but couldn’t really find the best answer. For one of our new websites we are thinking about the structure of this website and the corresponding URL-structure. Basically we have a main product (and a few main keywords) which should drive the most traffic to our website, and for which we want to optimize our homepage. Besides those main keywords, we have an enormous base of long-tail keywords from which we would like to generate traffic. This means we want to create a lot of specific pages which are optimized. My main question is the following: We are thinking of two options: Option 1: www.example.com/example-keyword-one Option 2: www.example.com/directory/example-keyword-one With option 1 we will link directly from our homepage to the most important pages (which represent our most important keywords). All the pages with the long tail content will be linked from another section on our website, which is one click away from our homepage (specifically a /solutions page which is linked from the footer). All the pages with long-tail content will have this structure www.example.com/example-keyword-one so the URLs will not contain the directory /solutions With option 2 we will use more subdirectories in our URLs. Specifically, for all the long tail content we would use URLs like this: www.example.com/solutions/example-keyword-one
Web Design | | NielsB
The directories we want to use wouldn't really have added value in terms of SEO, since they don’t represent important keywords. So what is the best way to go? Option 1, straightforward, short URL’s which don’t really represent the linking structure of our website, but only contain important keywords. Or option 2, choose for more directories in our URLs which represent the linking structure of our website, but contain directories which don’t represent important keywords. Would the keyword ‘solutions’ in the directory (which doesn’t really relate to the content on the page) have a negative impact on our rankings for that URL?0 -
Any reaction to the announcement from Google that 'signed in' searches won't pass through search query info to analytics?
Seems like SEO is about to get that much harder: http://analytics.blogspot.com/2011/10/making-search-more-secure-accessing.html Any thoughts on this?
Web Design | | PaulM011 -
How do search engines interpret <hgroup>...</hgroup> tags?
Hi there. I'm building an HTML 5 site and through research of new HTML 5 elements I've seen little conclusive information about the interpretation of the new <hgroup>tag, in terms of SEO application and interpretation. In particular does Google interpret the nested heading tags as individual elements or does it combine them into one entity? For example, say I have: <hgroup> # Article Heading ## Article Sub-heading </hgroup> How is this interpreted by Google and what would be some good SEO best practices regarding the <hgroup>tag in HTML5: Is it interpretted as a single tag (" Article Heading: Article Sub-heading ") or two separate heading tags (one and one )? Also, how much does the ordering of the tags matter (say for example I wanted something like the following for visual purposes? <hgroup> ## Article Sub-heading # Article Heading </hgroup> One last thing: is it safe to assume that it is indeed OK to have multiple tags on a single page (as referenced by Matt Cutts a while back in a Webmaster Video)? Thanks! </hgroup> </hgroup>
Web Design | | LMDNYC2 -
The primary search keywords for our news release network have dropped like a rock in Google... we are not sure why.
Hi, On April 11th, a month after the farmer update was released for U.S. users of Google, the primary keywords for ALL our sites significantly dropped in Google. I have some ideas why, but I wanted to get some second opinions also. First off, I did some research if Google did anything on the 11th of April... they did. They implemented the farmer update internationally, but that does not explain why our ranks did not drop in March for U.S. Google users... unless they rolled out their update based on what site the domain is registered in... in our case, Canada. The primary news release site is www.hotelnewsresource.com, but we have many running on the same server. EG. www.restaurantnewsresource.com, www.travelindustrywire.com and many more. We were number 1 or had top ranks for terms like ¨Hotel News¨, ¨Hotel Industry¨, ¨Hotel Financing¨, ¨Hotel Jobs¨, ¨Hotels for Sale¨, etc... and now, for most of these we have dropped in a big way. It seems that Google has issued a penalty for every internal page we link to. Couple obvious issues with the current template we use... too many links, and we intend to change that asap, but it has never been a problem before. The domain hotelnewsresource.com is 10 years old and still holds a page rank of 6. Secondly, the way our news system works, it´s possible to access an article from any domain in the network. E.G. I can read an article that was assigned to www.hotelnewsresource.com on www.restaurantnewsresource.com... we don´t post links to the irrelevant domain, but it does sometimes get indexed. So, we are going to implement the Google source meta tag option. The bottom line is that I think we put too much faith in the maturity of the domain... thinking that may protect us... not the case and it´s now a big mess. Any insight you can offer would be greatly appreciated. Do you think it was farmer or possibly something else? Thanks, Jarrett
Web Design | | jarrett.mackay0