Need Help On Proper Steps to Take To De-Index Our Search Results Pages
-
So, I have finally decided to remove our Search Results pages from Google. This is a big dealio, but our traffic has consistently been declining since 2012 and it's the only thing I can think of.
So, the reason they got indexed is back in 2012, we put linked tags on our product pages, but they linked to our search results pages. So, over time we had hundreds of thousands of search results pages indexed.
By tag pages I mean:
Keywords: Kittens, Doggies, Monkeys, Dog-Monkeys, Kitten-Doggies
Each of these would be linked to our search results pages, i.e. http://oursite.com/Search.html?text=Kitten-Doggies
So, I really think these pages being indexed are causing much of our traffic problems as there are many more Search Pages indexed than actual product pages. So, my question is... Should I go ahead and remove the links/tags on the product pages first? OR... If I remove those, will Google then not be able to re-crawl all of the search results pages that it has indexed? Or, if those links are gone will it notice that they are gone, and therefore remove the search results pages they were previously pointing to?
So, Should I remove the links/tags from the product page (or at least decrease them down to the top 8 or so) as well as add the no-follow no-index to all the Search Results pages at the same time?
OR, should I first no-index, no-follow ALL the search results pages and leave those tags on the product pages there to give Google a chance to go back and follow those tags to all of the Search Results pages so that it can get to all of those Search Results pages in order to noindex,. no follow them? Otherwise will Google not be able find these pages?
Can someone comment on what might be the best, safest, or fastest route?
Thanks so much for any help you might offer me!!
Craig
So, I wanted to see if you have a suggestion on the best way to handle it? Should I remove the links/tags from the product page (or at least decrease them down to the top 8 or so) as well as add the no-follow no-index to all the Search Results pages at the same time?
OR, should I first no-index, no-follow ALL the search results pages and leave those tags on the product pages there to give Google a chance to go back and follow those tags to all of the Search Results pages so that it can get to all of those Search Results pages in order to noindex,. no follow them? Otherwise will Google not be able find these pages?
Can you tell me which would be the best, fastest and safest routes?
-
Hi Craig,
In general - the structure looks ok - just wondering how you going to manage to keep 1mio products a reasonable number of clicks from the homepage.
rgds
Dirk
-
Sounds good! Thanks again!
C
-
Hi Craig,
Getting quite late here in Belgium (already past midnight) - will get back to you tomorrow (with a fresher mind...)
Dirk
-
This is a big help as I am finalizing the category pages now.
So our site is big, getting close to 1,000,000 products in the store.
Each product can belong to up to 3 sub-cats. Our internal category structure is generally like this:
Widgets->Awesome Widgets->Blue Widgets
or
Widgets->Awesome Widgets->Large Widgets->Large Blue Widgets
So, currently, my structure is like this:
1. Home Page Links To:
Primary Category 1
Primary Category 2
Primary Category 3
Primary Category 42. Each Primary Category Page:
1. Links any sub-categories
2. Has a list of all products in that category with pagination linking to their product pages.3. The Product Page Links back to:
1. Primary Category Page
2. Each of the 3 Sub-Categories' Pages that Product Belongs To.
3. A small number of related products.Generally each sub-cat will have thousands if not tens of thousands of sub-products.
How does this sound and do you have any advice related to this?
Thanks again!! :):):):):):):):) You get extra smilies for awesome help.
Craig
-
Hi Craig,
A. The logic seems ok - but doesn't say much about the depth of the site. Questions for me are:
- can one product belong to more than one category?
- are we talking about 100 products or 10.000?
Suppose worst case
- each product belongs to only one subcategory & each subcategory belongs to one category
- you have 500 products in this subcategory
If there is pagination - with 50 products/page the last 50 products will be >10 clicks from the homepage
If there a 'show as one page - there would be too many links on the page so you cannot be certain that the ones at the bottom of the pages will get followed.
If a product can belong to more subcategories or categories and/or there are fewer products, it's more likely that it will be closer to the homepage.
B. No - the products would not be removed from the index. However, if there are no links to these pages, they will not be shown in the results (google wants that each part of your content should be reachable by at least 1 link). No (internal) links = no value is the way Google thinks. The more links & the fewer clicks from the homepage the more value a page gets. You should put the new navigation in place as soon as possible - ideally it should have been done at the same time.
Hope this clarifies,
Dirk
-
I was talking about my search pages specifically, either adding a meta robots no-index,no-follow OR just a no-index. I just went ahead and added no-follow.
So, good point on the screaming frog.
Currently, the site is organized like this: HomePage -> Several links to many variations of the Search Page -> Product Pages
The new organization will be:
Home Page -> Various Category Pages -> Various Sub-Category Pages (With products on them and pagination to show all products) -> Possibly Other Sub-Category Pages (With products on them and pagination)
Then on the product pages there will be links back to the primary and secondary category pages.
A. How does that sound and
B. So, if I have Product pages that are already indexed could no-indexing the Search pages mean these pages get removed? Or, if they are already in the index, are they safe?
Thanks again for taking the time to help and answer!!
Craig
-
Hi Craig,
Not sure where you would put the nofollow:
-
the links to the search pages on the articles need to be of type "follow" - if Google is never allowed to follow the links to the search pages it will take a lot of time before the bot discovers that all the search pages became "noindex"
-
the links on the search pages themselves- here you can do what you want. As the final goal is to remove the search pages from the index - once they're not longer indexed it becomes irrelevant if the links on these pages are nofollow or not. I would keep these links of type "follow" - allowing the bots to easily access all the pages - find the links on them that go the other search pages and take them out of the index.
One thing that you should also check and that I didn't mention before - it is probably a good idea to crawl your site now with Screaming Frog and check the depth of the site (%of articles at 1/2/3... clicks from the homepage). It could be possible that if you remove the "search" pages a larger part of your content moves deeper in the site - this could have a potential negative impact on the ranking of these articles. If this is the case - you could decide
- to keep some of the search pages (but noindex/follow)
- to increase cross linking between normal articles
- to add some new index pages (again noindex/follow)
(or a mix of these)
rgds,
Dirk
-
-
Hey Dirk,
I have one more follow-up on this if you don't mind. My SEO auditor said I should both no-index AND no-follow the search results pages.
This concerns me a little bit as I am concerned it may have a negative effect on my Product pages as I will have to make sure they will be found in another way, which I will do, but it will take time of course.
Any reason why you just suggested no-index and did not include the no-follow and do you have any other insight on that?
Thanks!
Craig
-
Thank you my brother...
Very much appreciate the time you took for some thorough answers here....
Very good stuff and VERY much appreciated.
I had a chat with my SEO auditor today and he suggested no-indexing, no following the search pages and in about 30 days remove the product page links.
So, I will likely do that.
Much appreciation to you - Craig
-
I don't think there is an easy route here - you will have to get rid of these indexed search pages in any case. Keeping this low quality pages will continue to hurt your site.
If you currently don't have the resources to do the 'ideal' scenario - I would go for the short pain: cut out these pages now, it will probably cost you traffic on the short term, but at least you have a clean base to build upon. Keeping the pages is probably better on the short term, but the longer you keep them, the more your site's reputation is going to be affected and put's you in danger for future algorithm updates.
Just my opinion
Dirk
-
Right, I hear you on that, and honestly, that scenario you have posited, is the reason I haven't done anything yet on this. I agree that is the ideal way to do it, but I am not sure I can. I just don't have the time or resources and I agree that the positive effect could take some time...
So, I am curious, what you think the quickest route to a positive effect would be?
C
-
Hi,
There is an alternative solution but it would require more work on your side.
The problem with your current situation is that you create thousands of low value pages with little added value (which Google doesn't really like: https://www.mattcutts.com/blog/search-results-in-search-results/) and then you heavily promote these low quality pages by point hundreds of links to them. Principal message to Google - these low quality pages are my most important ones.
What you could do is to check the search pages which are generating traffic (ex. take the top 100) and create "real" pages for them. If we take the example you give: http://oursite.com/Search.html?text=Kitten - rather than having a generic search page with little added value you create a real page with some added value content (yoursite.com/topics/kitten) with links to your most important pages on the subject. As an example of how such a page could look like: http://dogtime.com/dog-breeds/german-shepherd-dog - this page is like a kind of "home" - containing a definition + links to the most important related articles on the subject. If these kinds of pages already exist on your site then of course no need to create them.
On the related search pages you then put a canonical url pointing to this page. You also update the links to the search page to the "real" added value page. This way you start promoting new value added content with minimal risk of loosing your current positions & remove the old low value pages from the index. It can take some time however before you see a positive effect.
For the search request where it's not possible to create a version with add value - you point the canonical to the generic search page (or your homepage) and remove all the links to these pages.
Hope this helps,
Dirk
-
Dirk,
THANKS!!! Thanks for the solid response. I guess my only concern is, we are still getting traffic from these indexed Search pages... and I need to minimize the hit from removing them. Any other more advanced methods I could use? Or.... In that case, would you recommend I do a combination of using the URL removal tool PLUS removing the tags?
I just need to do this as right as possible. I can't afford too much of a hit here (if any.) But, at the same time, we are losing traffic so fast, and have lost so much traffic, I don't have any choice at this point. We have doubled our product pages in the past 3 years and yet have lost about half our traffic.
Thanks again!
Craig
-
Hi,
I would first put a noindex on all your search result pages and leave the tags on the pages to allow Google to crawl them & "read" the new instructions.
I would also try to block these result pages in the robots.txt - it accepts pattern-matching ( https://support.google.com/webmasters/answer/6062596?hl=en&ref_topic=6061961) - if you try this make sure that you test it properly to avoid unwanted side effects.
You could also try the url removal tool - it's quite easy to delete an entire directory with the tool (https://support.google.com/webmasters/answer/1663419?hl=en) - you must make sure however that the pages cannot be crawled again (so do it after the modification of the robots.txt). If your search is on the root of your site and not in a separate directory, not sure if it's going to work.
Just removing the links to these pages without other modification is not going to help - they will just remain in the index.
Hope this helps,
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which pages should I index or have in my XML sitemap?
Hi there, my website is ConcertHotels.com - a site which helps users find hotels close to concert venues. I have a hotel listing page for every concert venue on my site - about 12,000 of them I think (and the same for nearby restaurants). e.g. https://www.concerthotels.com/venue-hotels/madison-square-garden-hotels/304484 Each of these pages list the nearby hotels to that concert venue. Users clicking on the individual hotel are brought through to a hotel (product) page e.g. https://www.concerthotels.com/hotel/the-new-yorker-a-wyndham-hotel/136818 I made a decision years ago to noindex all of the /hotel/ pages since they don't have a huge amount of unique content and aren't the pages I'd like my users to land on . The primary pages on my site are the /venue-hotels/ listing pages. I have similar pages for nearby restaurants, so there are approximately 12,000 venue-restaurants pages, again, one listing page for each concert venue. However, while all of these pages are potentially money-earners, in reality, the vast majority of subsequent hotel bookings have come from a fraction of the 12,000 venues. I would say 2000 venues are key money earning pages, a further 6000 have generated income of a low level, and 4000 are yet to generate income. I have a few related questions: Although there is potential for any of these pages to generate revenue, should I be brutal and simply delete a venue if it hasn't generated revenue within a time period, and just accept that, while it "could" be useful, it hasn't proven to be and isn't worth the link equity. Or should I noindex these "poorly performing pages"? Should all 12,000 pages be listed in my XML sitemap? Or simply the ones that are generating revenue, or perhaps just the ones that have generated significant revenue in the past and have proved to be most important to my business? Thanks Mike
Technical SEO | | mjk260 -
Large Drop in Indexed Pages But Increase in Traffic
We run a directory site and noticed about a week ago that Google Webmaster Tools was reporting a huge drop in indexed pages (from around 150,000 down to 30,000). In the same time, however, our traffic has increased. Has anyone seen this before or have any ideas on why this could happen? I have search for technical errors but nothing has changed on our site or our content.
Technical SEO | | sa_787040 -
I need help compiling solid documentation and data (if possible) that having tons of orphaned pages is bad for SEO - Can you help?
I spent an hour this afternoon trying to convince my CEO that having thousands of orphaned pages is bad for SEO. His argument was "If they aren't indexed, then I don't see how it can be a problem." Despite my best efforts to convince him that thousands of them ARE indexed, he simply said "Unless you can prove it's bad and prove what benefit the site would get out of cleaning them up, I don't see it as a priority." So, I am turning to all you brilliant folks here in Q & A and asking for help...and some words of encouragement would be nice today too 🙂 Dana
Technical SEO | | danatanseo0 -
Some competitors have a thumbnail in Google search results
I've noticed that a few of my top competitors have a small photo (thumbnail) next to their listing. I'm sure it's not a coincidence that they are ranked top for the search phrase too. Is this really a help and how can it be done? Many thanks, Iain.
Technical SEO | | iainmoran0 -
Page Not Found Help!
Hi, I recently (about 2 months ago) moved a blog from a separate domain name over to my eCommerce site to help with marketing. http://www.moondoggieinc.com/blog. I seem to have gotten it all to work right, but I'm getting tons of 404 errors and they all have " in them for example: http://www.moondoggieinc.com/blog/”http://www.moondoggieinc.com/custom_dog_tanks_and_tees.php” I'm not sure how this happened of how to fix it, but there are about 250 pages like this. I know how to redirect them all with a 301 in htaccess, but Im not sure if that's the appropriate course to fix this or if that's just putting a patch on something that's causing a more major issue. Or do i just need to write 250 301 redirects? Thanks! Kristy O
Technical SEO | | KristyO0 -
Need help with Joomla duplicate content issues
One of my campaigns is for a Joomla site (http://genesisstudios.com) and when my full crawl was done and I review the report, I have significant duplicate content issues. They seem to come from the automatic creation of /rss pages. For example: http://www.genesisstudios.com/loose is the page but the duplicate content shows up as http://www.genesisstudios.com/loose/rss It appears that Joomla creates feeds for every page automatically and I'm not sure how to address the problem they create. I have been chasing down duplicate content issues for some time and thought they were gone, but now I have about 40 more instances of this type. It also appears that even though there is a canonicalization plugin present and enabled, the crawl report shows 'false' for and rel= canonicalization tags Anyone got any ideas? Thanks so much... Scott | |
Technical SEO | | sdennison0 -
Non-Canonical Pages still Indexed. Is this normal?
I have a website that contains some products and the old structure of the URL's was definitely not optimal for SEO purposes. So I created new SEO friendly URL's on my site and decided that I would use the canonical tags to transfer all the weight of the old URL's to the New URL's and ensure that the old ones would not show up in the SERP's. Problem is this has not quite worked. I implemented the canonical tags about a month ago but I am still seeing the old URL's indexed in Google and I am noticing that the cache date of these pages was only about a week ago. This leads me to believe that the spiders have been to the pages and seen the new canonical tags but are not following them. Is this normal behavior and if so, can somebody explain to me why? I know I could have just 301 redirected these old URL's to the new ones but the process I would need to go through to have that done is much more of a battle than to just add the canonical tags and I felt that the canonical tags would have done the job. Needless to say the client is not too happy right now and insists that I should have just used the 301's. In this case the client appears to be correct but I do not quite understand why my canonical tags did not work. Examples Below- Old Pages: www.awebsite.com/something/something/productid.3254235 New Pages: www.awebsite.com/something/something/keyword-rich-product-name Canonical tag on both pages: rel="canonical" href="http://www.awebsite.com/something/something/keyword-rich-product-name"/> Thanks guys for the help on this.
Technical SEO | | DRSearchEngOpt0 -
/index.php in sitemap? take it out?
Hi Everyone, The following was automatically generated at xml-sitemaps.com Should I get rid of the index.php url from my sitemap? If so, how do I go about redirecting it in my htaccess ? <url><loc>http://www.mydomain.ca/</loc></url>
Technical SEO | | RogersSEO
<url><loc>http://www.mydomain.ca/index.php</loc></url> thank you in advance, Martin0