Search directory - How to apply robots
-
Hi.
On the site I'm working on, we use a search directory to display our search results. It displays as follows -
With the dynamic search results appearing after the hash tag.
Because of the structure of the website, many of the lefthand nav defers back to this directory. I know that most websites "noindex, nofollow" the search results pages, but due to the ease of customers generating them, I'm afraid that if I do this, we'll miss out on the inevitable links customers will provide...and, even though it's just the main search directory, these links will still help my domain. The search is all java-generated so there's nothing for spiders to follow within this directory - save the standard category nav.
How should I handle this?
Thanks.
-
That was my first instinct as well - but I wanted to check if there was a better move? Thanks for your help!
-
Have you considered switching to simply "noindex, follow" for these pages? This would both keep them out of the search index and would allow for the benefit of external links to these pages.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Disallow: /sr/ and Disallow: /si/ - robots.txt
Hello Mozzers - I have come across the two directives above in a robots.txt file of a website - the web dev isn't sure what they meant although he implemented robots.txt - I think just legacy stuff that nobody has analysed for years - I vaguely recall sr means search request but can't remember. If any of you know what these directives do, then please let me know.
Web Design | | McTaggart0 -
Is there anything wrong to have large number of internal links pointing to homepage? Including links from sub domains or sub directories?
Hi all, Generally more number of internal links will be pointed to homepage. But I see some modern suggestions that too many internal links to homepage are not good. I'm just wondering if most number of internal links pointing to homepage may hurt? Also we have sub domains, can we point a link from every page of sub domain or sub directory to homepage? Usually the answer here is about users. Of course, the content is about same product across all pages. Thanks
Web Design | | vtmoz0 -
Do search engines see copy/keywords when it appears only at the bottom of a page?
My client is looking to improve their SEO, and to date I've written meta data and made some initial recommendations. Thing is, on some of their pages, the body copy appears at the bottom of the page, past links and big, splashy images. My question is, will search engines even see that copy to crawl it for keywords? Thanks!
Web Design | | MarcieHill0 -
Google Search Console Block
Am new to SEO. My clients site was completed using Yoast premium and then used Google search console to initiate the crawl. Initially setup an http:// property and all seemed good. Then i removed that under search console an created an https:// did the render and it appears google has put a block and placed their own robots.txt file which basically has rendered the site useless. Feedback most appreciated.
Web Design | | BoostMyBiz0 -
Any second opinions as to why our organic search website traffic hasn't recovered from website rebrand (domain change, website redesign)?
I am hoping to see if anyone in the Moz community would be able to help troubleshoot or lend any advice on a major organic search traffic issue we've been experiencing over the last 8 months. In a nutshell, we decided our ~4.5-year-old business needed to undergo a rebrand in October 2015. After changing domains & redesigning our website (more below), our search-driven sessions have dropped 20% in 2016 v.s. 2015. We made quite a few on-site modifications (with some success) post-redesign but are still deep in a rut and not sure what more we can do to recover. I've listed my theories below as to why we're still suffering this hit. If anyone could weigh in on these and/or share any other troubleshooting ideas, I would greatly, greatly appreciate it (and owe you a lunch/beverage of your choice the next time I'm in your city!). ****Backlinks - despite our efforts to 301 all links, I sense we have lost many backlinks. According to Open Site Explorer, our old domain has 1,172 backlinks (some from some very authoritative pages domains), 1,068 of which are passing link equity. In contrast, our new domain has 367 backlinks, 321 are passing link equity, and very few overlap with our old domain. Domain Age - we may have lost much of our reputation with Google as our new domain is much younger than our old domain (1-year-old v.s. 5.5 years old). Domain Name - although I thought to have common keywords in one's domain was a myth, I am now questioning that belief. Our old domain contained a popular, topical keyword and our new domain is derived from a term that is topical, but very uncommon. New URLs - our developer has insisted all links were moved to the new domain, but I have a hunch they were not. When conducting a "site search" (i.e. "site:websitename.com"), the new domain returns 7,740 results. Prior to our switch, a site search with the old domain yielded 30,000+ results. 404s - we found and fixed 100-200 404'd links after the domain switch. We still see a few pop-up today and I'm wondering if this is a red flag in Google's eyes. For a little more background too, here are the nitty gritty details with a rough timeline: Pre-October 12, 2015 - registered new domain and designed the new website on Wordpress, while researching a range of articles and resources for a successful site migration (e.g. this and this Moz guide). October 12, 2015 - flipped the switch on the website design, domain, minor content reorganization, and social handles. We announced the change to our audience via an article, newsletter, and social; informed Google Webmaster Tools (GWT) of the new address, 301'd all links from the old to the new domain, and submitted new sitemap in GWT. October 12 - 16, 2015 - traffic is normal, everything seems to be okay. October 17, 2015 - search traffic drops by 54% v.s. the same day of week pre-rebrand. October 26, 2015 - search traffic rises, so now only down by 30% v.s. the same day of week pre-rebrand. November/December 2015 - re-added numerous elements from the old website such as category, tag, and page pagination and a few sidebar modules that linked to other important pages and tags. Search traffic rises slightly in November (down 27% year-on-year), dips again in December (down 31% year-on-year). January 2016 - today (June 17, 2016) - we published more content on a daily basis and search traffic fluctuates around the 20% versus the same period in 2015. January 2016 - down 23% year-on-year February 2016 - down 17% year-on-year March 2016 - down 20% year-on-year April 2016 - down 21% year-on-year May 2016 - down 21% year-on-year June 2016 (until the 17th) - down 23% year-on-year Thank you all in advance for your time and help, please let me know if you have any questions!
Web Design | | nick490 -
Robots.txt being blocked
I think there is an issue with this website I'm working on here is the URL: http://brownieairservice.com/ In Google Webmaster tools I am seeing this in the Robots.txt tester: User-agent: *
Web Design | | SOM24
Crawl-delay: 1
Disallow: /wp-content/plugins/
Disallow: /wp-admin/ Also when I look at "blocked resources" in the webmaster tools this is showing to be blocked: http://brownieairservice.com/wp-content/plugins/contact-form-7/includes/js/jquery.form.min.js?ver=3.51.0-2014.06.20It looks like the form plug in is giving the issues but I don't understand this. There are no site errors or URL errors so I don't understand what this crawl delay means or how to fix it. Any input would be greatly appreciated. Thank you0 -
Can you recommend a responsive web design company of template to accommodate mobile searches
I need to change my old Dreamweaver website www.endeavourcottage.co.uk into a responsive website that can be viewed and used easily on mobile phones. Four years ago when the site was originally built it was one cottage on one basic website but it's grown into a small letting agency with several properties, so I think it's time to invest in a fully functional website. Looking at Google analytics around the fifth of the searches and now done on a mobile phone each month on this particular site, and obviously that's going to become a larger percentage in the future. Does anybody know of a particular company that can produce good quality responsive websites? Or alternatively used a particular responsive website template which was good. One of the guys that work for me is quite a good web designer if we got the right template. The old website is grown into quite a large beast with a blog so I think the idea would be to change the main pages with the properties on, just linking back to the old websites quite extensive blog / historical interest pages which I've built over the last few years. I need to keep the old blog webpages indexed because they have hundreds of back links coming with longtail keywords producing traffic. Have you gone through the process of changing your main business website for a resonsive redesign recently and how did you approach it?
Web Design | | whitbycottages0 -
Any reaction to the announcement from Google that 'signed in' searches won't pass through search query info to analytics?
Seems like SEO is about to get that much harder: http://analytics.blogspot.com/2011/10/making-search-more-secure-accessing.html Any thoughts on this?
Web Design | | PaulM011