On-site Search - Revisited (again, *zZz*)
-
Howdy Moz fans!
Okay so there's a mountain of information out there on the webernet about internal search results... but i'm finding some contradiction and a lot of pre-2014 stuff. Id like to hear some 2016 opinion and specifically around a couple of thoughts of my own, as well as some i've deduced from other sources. For clarity, I work on a large retail site with over 4 million products (product pages), and my predicament is thus - I want Google to be able to find and rank my product pages. Yes, I can link to a number of the best ones by creating well planned links via categorisation, silos, efficient menus etc (done), but can I utilise site search for this purpose?
-
It was my understanding that Google bots don't/can't/won't use a search function... how could it? It's like expeciting it to find your members only area, it can't login! How can it find and index the millions of combinations of search results without typing in "XXXXL underpants" and all the other search combinations? Do I really need to robots.txt my search query parameter? How/why/when would googlebot generate that query parameter?
-
Site Search is B.A.D - I read this everywhere I go, but is it really? I've read - "It eats up all your search quota", "search results have no content and are classed as spam", "results pages have no value"
I want to find a positive SEO output to having a search function on my website, not just try and stifle Mr Googlebot. What I am trying to learn here is what the options are, and what are their outcomes? So far I have -
_Robots.txt - _Remove the search pages from Google
_No Index - _Allow the crawl but don't index the search pages.
_No Follow - _I'm not sure this is even a valid idea, but I picked it up somewhere out there.
_Just leave it alone - _Some of your search results might get ranked and bring traffic in.
It appears that each and every option has it's positive and negative connotations. It'd be great to hear from this here community on their experiences in this practice.
-
-
Hopefully that helps you some I know we ran into a similar situation for a client. Good luck!
-
Great idea! This has triggered a few other thoughts too... cheers Jordan.
-
I would recommend using screaming frog to crawl only product level pages and export them to a csv or excel doc then copy and past your xml sitemap into an excel sheet. Then from there I would clean up the xml sitemap and sort it by product level pages and just compare the two side by side and see what is missing.
The other option would be to go into google webmaster tools or search console and look at Google Index -> index status and then click the advanced tab and just see what is indexed and what all is being blocked by the robots.txt.
-
@jordan & @matt,
I had done this, this was my initial go-to idea and implementation, and I completely agree this is a solution.
I guess I was hoping to answer the question "can Google even use site search?". as this would answer whether the parameter even needs excluding from robots.txt (I suspect they somehow do, as there wouldn't be this much noise about it otherwise).
That leaves the current situation - Does restricting google from searching my internal search results hinder it's ability to find and index my product pages? I'd argue it does, as since implementing this 6 months ago, the site index status has gone from 5.5m to 120k.
However, this could even be a good thing, as it lowers the Googlebot activity requirement, and should focus on the stronger pages... but the holy grail I am trying to achieve here is to get all my products indexed so I can get a few hits a month from each, i'm not trying to get the search results indexed.
-
Agree with Jordan - block the parameter for search in robots.txt and forget it. It won't bring search traffic in, it shouldn't get crawled but if it does, it's always a negative.
-
I cant speak for everyone but generally we like to robots.txt the search pages. I would imagine since you are working on a large retail site you would want to ensure your other pages get indexed properly so I would imagine blocking the search pages with a robots.txt would suffice. I would also look for some common reoccuring searches through the site search to possibly build content around as well.
I hope that helps some.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site migration/ CMS/domain site structure change-no access to search console
Hi everyone, We are migrating an old site under a bigger umbrella (our main domain). As mentioned in the title, We'll perform CMS migration, domain change, and site structure change. Now, the major problem is that we can't get into google search console for the old site. The site still has old GA code, so google search console verification using this method is not possible, also there is no way developers will be able to add GTM or edit DNS setting (not to bother you with the reason why). Now, my dilemma is : 1. Do we need access to old search console to notify Google about the domain name change or this could be done from our main site (old site will become a part of) search console 2. We are setting up 301 redirects from old to the new domain (not perfect 1:1 redirect ). Once migration is done does anything else needs to be done with the old domain (it will become obsolete)? 3.The main site, Site-map... Should I create a new sitemap with newly added pages or update the current one. 4. if you have anything else please add:) Thank you!
Intermediate & Advanced SEO | | bgvsiteadmin0 -
Researching search volume drop
I am seeing a pretty precipitous drop in search volume traffic (see link). My keyword rankings don't seem to have suffered too much over this period. In fact, my #1 keyword have actually increased slightly in this timeframe. Two questions... Is there some way to assess overall search volume across my tracked keywords (to see if this is just a case of overall searches dropping)? Is there a recommended plan of attack for investigating drops like this - beyond overall search volume, what other data might be important in identifying the cause of this. In short, I'm looking for some logic/structure for how I investigate this, using Moz tools and reports. Thanks. Mark omE1VPc
Intermediate & Advanced SEO | | MarkWill0 -
Meta refresh for news site?
We have a news site that uses a meta refresh (<meta < span="">http-equiv="refresh" content="600" /> across all content. I understand the reasoning on the homepage and am trying to decide of the cons of using this (slows page, is it treated differently and loses pagerank/link ..) Does anyone have experience with meta refresh being a negative thing or does it no longer matter?</meta <>
Intermediate & Advanced SEO | | KristieWahlquist0 -
How is my 301 redirected site stealing rankings from the main site?
Hello, I have a site, drhobelt.com, that 301 redirects to the main site, drhonow.com. Not only is drhobelt.com still indexed, but it recently stole rankings from drhonow.com for "decompression belt" related terms. What could be causing this? How do I reclaim the rankings for drhonow.com? Thanks for reading!!
Intermediate & Advanced SEO | | DA20130 -
Why has my site dropped off the face of Google???
Hi All, My site was ranking well for a long time and suddenly can't be seen at all any more. I have been trying to figure this out for some time and can't get to the bottom of it. Funny thing is also even when searching for my site with a keyword "snowboard gulmarg" my URL www.klinehinmalaya.com does not appear and somewhere way back in the listing my another page www.klinehimalaya.com/packages.php comes up. Any help would be good right now. Thanks in advance, Catherine.
Intermediate & Advanced SEO | | caherinechan0 -
Migrating a very large site
hello, we are thinking about changing imageworksstudio.com to imageworkscreative.com we have TONS of of pages and want to make sure that our Page Rank and Rankings are maintained. Are there any best practices or specific guides for redirecting for such a large site (which already has a bunch of redirects in place in the first place) Thanks!
Intermediate & Advanced SEO | | imageworks-2612900 -
2 sites or one sites: 2 locations
Hello, I have a dog training client who is offering services in 2 separate locations. We're looking to be first in the non-local search results and also rank well in google places. Would it be better to go for 2 separate sites or one site and try to rank for 2 different locations with one site? There's both local and standard search results when we type in our keywords. Thanks!
Intermediate & Advanced SEO | | BobGW0 -
Dupicated Site Issues?
We are launching a new site for the Australian market and the URL will just be siteAU.com. Currently the tech team (before we came on board) has it setup with almost exactly the same content (including the site css/nav/structure etc). Some product page content is slightly different, and category pages have different product orders, plus there are location pages that are specific to AU, but otherwise it's the same. The original site: site.ca has been around for 6+ years, with several thousand pages and solid organic ranking (though the last few months have dropped ) Will the new AU site create issues for the original domain? We also have siteUSA.com which follows the same logic and has been live for a while.
Intermediate & Advanced SEO | | BMGSEO0