Culling 99% of a website's pages. Will this cause irreparable damage?
-
I have a large travel site that has over 140,000 pages. The problem I have is that the majority of pages are filled with dupe content. When Panda came in, our rankings were obliterated, so I am trying to isolate the unique content on the site and go forward with that.
The problem is, the site has been going for over 10 years, with every man and his dog copying content from it. It seems that our travel guides have been largely left untouched and are the only unique content that I can find. We have 1000 travel guides in total.
My first question is, would reducing 140,000 pages to just 1,000 ruin the site's authority in any way?
The site does use internal linking within these pages, so culling them will remove thousands of internal links throughout the site.
Also, am I right in saying that the link juice should now move to the more important pages with unique content, if redirects are set up correctly?
And finally, how would you go about redirecting all theses pages? I will be culling a huge amount of hotel pages, would you consider redirecting all of these to the generic hotels page of the site?
Thanks for your time, I know this is quite a long one,
Nick
-
Thank you all for the positive feedback.
Lately I have made the time for SEOmoz Q&A as I have been doing various SEO research and these boards can be a great way to stretch thought processes.
-
Seriously, Ryan is always ALL OVER Seomoz comments with good feedback
-
Just figured out how to do this, I'm new to SEOMoz Q&A, thanks for the nudge! Ryan certainly deserves it!
-
I do hope Ryan gets "Good Answer" and/or "Endorsed Answer" for this... hint, hint
-
Your understanding is correct.
Google does not care how many directories appear in a URL. The two URLs you offered as an example are viewed equally by Google. What's important is how many clicks it takes users to access those links.
-
Hi Ryan,
Sorry for not getting back to you straight away, I've been in meetings all day.
You've given me some excellent ideas again!!
Just to clarify, the old URL's are in the following format:
www.url.com/resort_hotels/hotels_in_rome.asp
I am aiming to use the following structure for the new website:
or
I was wondering if you knew from a search engine perspective, which URL is the better option. From a user perspective, I would assume the second.
I am operating under the assumption that Google rates a URL's importance by the number of clicks it is from the homepage and not the number of directories (www.url.com/.../.../...) within the URL?
If this is the case I will probably go for the second URL structure, but place links higher up the hierarchical structure of the site for the more important locations.
Unfortunately, the landing pages for the cars and flights house exactly the same content with just the location text tweaked. There is nothing else unique on these pages, which is why I find myself with no other option but to get rid of them.
I really like your idea of testing landing pages for a specific area. This may be a good way to go, but creating two paragraphs of text for both the flight and car hire pages is not an option at this time. With 40,000 locations we’d need to produce 160,000 paragraphs of unique text, which would cost around $400,000, may be slightly less with bulk discounting.
If I was to spend that much money on content writing, I would probably expand the hotel side of the site as this is most profitable. But my priority after the launch of the new site is an extensive link building campaign to assist the transition.
Thanks so much again for your help Ryan, you're a star!
Did you know whether Google rates a URL's importance by the number of clicks it is from the homepage and not the number of directories (www.url.com/.../.../...) within the URL? It is really important that I find this one out!
Take care buddy,
Nick
-
Nick,
Sounds like you have a good strategy. I only have two additional items to share based on your latest reply.
www.url.com/resort_hotels/hotels_in_rome.asp
That url seems a bit spammy to me. Mentioning "hotels" twice is something I would avoid. I would consider something along the lines of the below options instead:
www.url.com/resorts/hotels_in_rome
www.url.com/resort_hotels/rome
I also wanted to talk about the landing pages for cars and air travel once more. Before directing all your current pages to a generic page I would take a look at the existing 140 pages and ask once again, do any of the pages have anything that is unique which can be used for the location based car and air landing pages?
Your plans are to develop these pages with quality content over time, which is great. I hate the idea of having establishing pages for each area, pulling back to having one generic page, then expanding again to location-based pages.
If you sincerely intend to develop these pages on a reasonable time period, I would suggest establishing one page for each location even if it was thin on content to start with. Driving directions, local driving laws, testimonials, anything that can be used as a starting point to hold your footing would be preferred.
If you do pull back to a generic "car rentals" page, I have two ideas. Build out your location landing page for one area such as London. Closely watch your conversion rates on users on the London page versus the generic page. If there is a significant difference, it may help speed up your transition. If you realize you are losing $$ every day you don't have those pages, then perhaps you can hire additional help to speed up the process.
The final idea would be to build country-based landing pages for car rentals as an stop-gap measure. Your Milan, Rome, etc pages could all direct to "Cars Italy" and "Air Italy".
There are tons of choices on the internet for travel providers. You have an extremely well established user base. My top concern for any migration is to maintain all my existing relationships. Some travel sites do great with a single landing page for air/cars/hotels. It sounds like your site has catered to clients in a specific way, and I would be sensitive to maintaining your current user experience.
One last idea that just came to me. After the migration poll users for feedback. Take surveys, offer discounts, generate hype but engage users because they will offer a different point of view which you may not have considered.
-
Ryan, you have given me some excellent ideas here and a great overall structure to make the transition between sites. I can't thank you enough for your help. I will certainly consult an SEO before proceeding with anything, but your insight has given me a lot to think about.
With regards to the sites current pages, the majority of locations only have 3 pages; Hotels, Car Hire & Flights. It is the amount of locations covered that make the site so expansive.
So with Hotels being our biggest earner, my idea going forward was to:
-
Use the travel guide's unique content for the hotel landing pages, i.e. [Hotels in Rome]
-
Redirect all of the old Car & Air location pages to the new website’s generic Car Hire & Flights pages.
This would mean that there wouldn’t be any location-based pages for Flights and Car Hire. The idea would be to build these up gradually as it would take some time and money to add the unique content required.
- From every hotel landing page we would use anchor text to promote the generic Flights and Car Hire pages. For example, [Buy Cheap Flights] or [Cheap Car Hire]
This additional anchor text should help our external link building and the generic Flights and Car Hire pages would house a search form for users to search any location.
So essentially, the majority of the site would be made up of Hotel landing pages, until we began building the site further.
I can see that your main concern is that the correct redirects are in place.
The site currently has the following URL structures, with locations for each:
Apart from the sitemaps, each have locations with them, for example:
www.url.com/resort_hotels/hotels_in_rome.asp
So my idea is to:
1) Redirect all “resort_hotels” URL’s to their relevant hotel page on the new website, for example,
www.url.com/resort_hotels/hotels_in_rome.asp
will go to the “Hotels in Rome” page on the new website.
- The rest of the pages will be redirected to the home page for their category, for example,
will go to “Flights” home page on the new website; and,
will go to “Car Hire” home page on the new website, etc.
Unless there is something really wrong with this strategy, or you have any instant criticism, I would like to thank you for your help again and ask that if you need anything, please don’t hesitate to drop me a message on here. You have given up enough your time and I’m more than grateful.
Kind Regards,
Nick
[I am using my work’s account, which is why I am displayed as Steve]
-
-
The transition I mentioned would allow for a smoother migration process rather then a "cold turkey" switch from the old site to the new site. You clearly recognize the end goal is to create your new site and delete the old site. The good news is that change does not have to happen over night.
You can build out your new site completely and go live with it. At that point you would update any external links you control along with your advertisements, signatures, etc. You would also want to reach out to partners and any sites with links that you can influence. Update those links so they point to your new pages.
The final step is the redirection of your 140k page old site to the appropriate pages on the new site. Clearly you wish to begin with the most prominent pages such as your landing pages along with any important pages such as "Contact Us", your reservation system, etc.
The next step would be applying your redirect rules to the remaining pages. Extensive testing will be required.
You should set up GA or another tracking tool to monitor your old site. You will want to closely monitor activity for quite some time. Specifically look for any issues with 404s and multiple redirects.
With respect to your anchor text, I suspect it was used to sculpt your site so your link value was focused on a particular page for each topic. When you have 140 pages on a given topic, you can pursue an incredible amount of longtail phrases. Now I suspect you may have 4 pages for each area: Rome, Rome by Air, Rome by Car, and Rome hotels. If that is the case your future anchor text linking will be a lot more straight forward.
I want to say "I wouldn't be concerned about the anchor text" but you have a major project ahead of you, you are highly dependent on SEO and there are many opportunities for something to go wrong. In that context, I would share the anchor text would be on the list of things to think about, but the proper redirects is a much larger concern.
A final thought I would offer: this is all high level, generic advice. I would recommend hiring a SEO who could offer a proper evaluation of your site along with a migration plan. Once the change has been completed and tested, you should gain many advantages with your new site. Hopefully they will offset any loss from the migration. Once you are confident in your new site, I would recommend a SEO campaign promoting your new site.
-
Hi again Ryan,
All the URL's are currently coded as .asp (www.url.com/Rome.asp) and we aim to build the new site with user friendly permalinks (www.url.com/Rome). So in answer to your question, yes, the sites could co-exist.
I'd hadn't thought of doing it this way, what a great idea.
With regards to the site's internal linking structure, I'm probably not explaining myself correctly. I understand that all of the site's juice needs to be recycled, but I'm now thinking that on many of the 120,000 pages there are links with anchor text to other relevant parts of the site, will removing these links, because there are so many of them, ruin the site's authority.
In addition, I would be really interested to hear your ideas on staging a transition.
I can't thank you enough for this Ryan, my head's spinning at the moment!
-
You are on the right track. The link value from your existing pages must be saved.
Prior to offering a further reply I would like to ask a couple questions:
-
how are your currently URLs coded? As an older site I presume your page URLs end in .asp?
-
will your new design also be in asp?
What I am trying to determine is, will the new site require new URLs. If your current page is /rome.asp and the new page will be /rome.php then the URL will change so both your new site and old site can co-exist at the same domain. This process will be helpful for staging a transition.
PS. My recommendation for URLs would be to use friendly URLs which do not show an extension (i.e. /rome) but that is not the present focus.
-
-
Thanks for a swift answer Ryan, very helpful indeed!
Put simply, the site is split into three key areas, Hotels, Flights & Car Rentals, each with about 40,000 pages each. The problem is that each of these pages uses a generic paragraph or two that is more or less the same, but tweaked slightly to update the location in question. For example,
"Our goal is to provide the best choice of hotels in Rome."...
"Our goal is to provide the best choice of hotels in Barcelona."...
Obviously Google sees this as duplicate content and rightly so, but other than rewriting 120,000 pages of content, I can't see an alternative to the problem, other than to remove the pages in question.
The site has so many quality links going into it, from authorities all over the web; it would be a shame to waste this juice on pages that are getting penalised.
The travel guide areas are all unique; there is a single guide for each of the 1000 destinations. For example,
http://www.url.com/guides/rome
My idea was to use this unique content to promote our hotel pages, for example,
http://www.url.com/hotels-in-rome
This page would have the unique travel content from that area plus a list of the hotels we have available in Rome.
Every other duped page on the site relating to "Rome", does have "Rome" in its URL, so a regex expression could be used to redirect all "Rome" themed pages to the "Hotels in Rome" page that would house the unique content.
All other pages that did not have unique content written about them could be redirected to the generic Hotels, Flights or Car Rentals pages as they all have either “hotels”, “flights” or “car-rental” in their URL.
The site is over 10 years old, is written in .asp and is managed with a bespoke piece of software created specifically for the site itself.
However, this doesn't really matter as we’re having the website redesigned at the same time as removing the dupe content and it will be built to our own specification.
My idea is to begin building up these locations from when the redesign goes live. This way I could keep a track of our content as it expands.
My main worry is that the culling of these pages, will remove 99% of our internal linking structure. And I'm wondering if removing this will dramatically reduce the authority of the site. However, at this point I’m struggling to see another option.
Sorry for the length of this reply, any ideas would be welcome, I just thought it would be best if you knew a bit more of the background.
Thanks again Ryan!
-
Hi Steve.
I would suggest taking a good look at your content pages. I understand having dupe pages, but you are suggesting a possible 140:1 ratio which is....wow.
As I am sure you are aware, there is not going to be any quick and easy fix. Here are some initial thoughts:
The first step I would take is to look for ANY commonalities between pages you can grab. For example, you mention 1000 travel guides. Are all the travel areas unique? For example, is there only one guide for Rome? If so, do all the duplicate Rome pages have "Rome" in the URL? If so, you can consider adding a regex expression to your htaccess file (presuming you are on an apache server) which could cover your 301s.
Is there any unique code on the pages which are common for a given guide? Using the above example, do all the "Rome" pages have "Rome" in the page title? If so, you could possibly update all the "Rome" pages by adding the correct canonical to the page's meta information. At this point it would truly depend on how your site is coded. Do you have a CMS? In what language are your pages written?
The bottom line, there is simply too much value in those pages to discard them. They each need to be properly 301'd as the preferred method. The 301s really need to be handled with general expressions which cover a large number of pages at once. You cannot use individual redirects even if you wanted to as it would cripple your web server.
I would not redirect all the pages to a single home page unless every other opportunity was completely explored.
-
Wow that's a big, bold move! I don't know how to answer it but if I were you I'd wait until you get a few, nice and comprehensive answers on here before doing anything to drastic. Either that or use a private Q&A question to SEOmoz staff if you have any points spare to do so. With such a large change, you want to ensure you're doing it right.
I'll be interested to see the answers you get for this.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Dynamic Pages with No Results Causing Thin Content
Hi Mozers, We have dynamic listing pages that pull in clinical trial results for specific disease types. Sometimes diseases have clinical trials and sometimes they don't. This means that sometimes the page will have zero results and sometimes it will return results. We have a sizable number of these so that when there are zero results, these pages look like thin content. What is the recommended method of dealing with this? Is there a way of doing a conditional noindex where the page is indexed if results are pulled in and and not indexed when the page returns zero results? If we can do this, should we? Will it confuse Google and send negative signals? Any guidance/thoughts are much appreciated! Yael
Intermediate & Advanced SEO | | yaelslater0 -
We're currently not using schemas on our website. How important is it? And are websites across the globe using it?
Schemas looks like an important thing when it comes to structuring your website and ensuring the crawl bots get all the details. I've been reading a lot of articles around the web and most of them are saying that schemas are important but very few websites are using it. Why so? Are the schemas on schema.org there to stay or am I wasting my time?
Intermediate & Advanced SEO | | Shreyans920 -
Don't affiliate programs have an unfair impact on a company's ability to compete with bigger businesses?
So many coupon sites and other websites these days will only link to your website if you have a relationship with Commission Junction or one of the other large affiliate networks. It seems to me that links on these sites are really unfair as they allow businesses with deep pockets to acquire links unequitably. To me it seems like these are "paid links", as the average website cannot afford the cost of running an affiliate program. Even worse, the only reason why these businesses are earning a link is because they have an affiliate program; that to me should violate some sort of Google rule about types and values of links. The existence of an affiliate program as the only reason for earning a link is preposterous. It's just as bad as paid link directories that have no editorial standards. I realize the affiliate links are wrapped in CJ's code, so that mush diminish the value of the link, but there is still tons of good value in having the brand linked to from these high authority sites.
Intermediate & Advanced SEO | | williamelward0 -
If it's not in Webmaster Tools, is it Duplicate Title
I am showing a lot of errors in my SEOmoz reports for duplicate content and duplicate titles, many of which appear to be related to capitalization vs non-capitalization in the URL. Case in point, if a URL contains a lower character, such as: http://www.gallerydirect.com/art/product/allyson-krowitz/distinct-microstructure-i as opposed to the same URL having an upper character in the structure: http://www.gallerydirect.com/art/product/allyson-krowitz/distinct-microstructure-I I am finding that some of the internal links on the site use the former structure and other links use the latter structure. These show as duplicate title/content in the SEOmoz reports, but they don't appear as duplicate titles in Webmaster Tools. My question is, should I try to work with our developers to create a script to change all of the content with cap letters in the destination links internally on the site, or is this a non-issue since it doesn't appear in Webmaster Tools?
Intermediate & Advanced SEO | | sbaylor0 -
Can I swap a website yet keep it's high ranking for a competitive keyword?
Couldn't fit the entire question in the main bit so the explanation is here: Working on a client's website which is hosted by volusion and also been doing SEO for them for about a year. Now we've finally got them ranking at the lower end of page 1 (around 10+) for their main keyword. They now want to move from volusion over to Amazon Web Store 😢 which seems to be an SEO nightmare from even my basic understanding of SEO. From looking at the coding and the way Amazon Web store is built on top of how restricted you are from doing anything with it, I am almost certain the shop will be extremely difficult to optimise and we will have to completely change nearly all of the content. Finally! the actual question; I was thinking I could get them to delay their move to Amazon webstore until they are ranking in the top 5 for this top keyword. Once they switch over, i assume they'll keep this ranking for at least a short while? This keyword attracts a high volume of traffic and if this traffic is clicking on the result for their website, and google sees that people are finding this website valuable (not clicking back onto google results). Will they be able hold onto this high ranking? Basically what I'm asking is, this will be a terrible outdated badly SEO'd shop, but if a high volume of people are clicking on it and staying on it from their lingering ranking will Google just let it stay at the top? A massive amount of gratitude in advance for anyone who tries to help with this! 😄
Intermediate & Advanced SEO | | acecream0 -
A Noob's SEO Plan of attack... can you critique it for me?
I've been digging my teeth into SEO for a solid 1.5 weeks or so now and I've learned a tremendous amount. However, I realize I have only scratched the surface still. One of the hardest things I've struggled with is the sheer amount of information and feeling overwhelmed. I finally think I've found a decent path. Please critique and offer input, it would be much appreciated. Step One: Site Architecture I run an online proofreading & editing service. That being said, there are lots of different segment we would eventually like to rank for other than the catch-all phrases like 'proofreading service'. For example, 'essay editing', 'resume editing', 'book editing', or even 'law school personal statement editing'. I feel that my first step is to understand how my site is built to handle this plan now, and into the future. Right now we simply have the homepage and one segment: kibin.com/essay-editing. Eventually, we will have a services page that serves almost like a site-map, showing all of our different services and linking to them. Step Two: Page Anatomy I know it is important to have a well defined anatomy to these services pages. For example, we've done a decent job with 'above the fold' content, but now understand the importance of putting the same type of care in below the fold. The plan here is to have a section for recent blog posts that pertain to that subject in a section titled "Essay Editing and Essay Writing Tips & Advice", or something to that effect. Also including some social sharing options, other resources, and an 'about us' section to assist with keyword optimization is in the plan. Step Three: Page Optimization Once we're done with Step Two, I feel that we'll finally be ready to truly optimize each of our pages. We've down some of this already, but probably less than 50%. You can see evidence of this on our essay editing page and proofreading rates page. So, the goal here is to find the most relevant keywords for each page and optimize for those to the point we have A grades on our on-page optimization reports. Step Four: Content/Passive Link Building The bones for our content strategy is in place. We have sharing links on blog posts already in place and a slight social media presence already. I admit, the blog needs some tightening up, and we can do a lot more on our social channels. However, I feel we need to start by creating content that our audience is interested in and interacting with them on a consistent basis. I do not feel like I should be chasing link building strategies or guest blog posts at this time. PLEASE correct me if I'm off base here, but only after reading step five: Step Five: Active Link Building My bias is to get some solid months of creating content and building a good social media presence where people are obviously interacting with our posts and sharing our content. My reasoning is that it will make it much easier for me to reach out to bloggers for guest posts as we'll be much more reputable after spending time doing step 4. Is this poor thinking? Should I try to get some guest blog posts in during step 4 instead? Step Six: Test, Measure, Refine I'll admit, I have yet to really dive into learning about the different ways to measure our SEO efforts. Besides being set up with our first campaign as an SEOPro Member and having 100 or so keywords and phrases we're tracking... I'm really not sure what else to do at this point. However, I feel we'll be able to measure the popularity of each blog post by number of comments, shares, new links, etc. once I reach step 6. Is there something vital I'm missing or have forgotten here? I'm sorry for the long winded post, but I'm trying to get my thoughts straight before we start cranking on this plan. Thank you so much!
Intermediate & Advanced SEO | | TBiz2 -
Understanding Google's keyword tool...
When I type in Google a keyword like : boot camp I get results that show Bootcamp (one word) traffic in the tens of thousands. I see many words combined. Does this mean that tens of thousands of people every month are misspelling that keyword? How should I interpret this in terms of anchor texting? I would hate to deliberately misspell it on my website just to get traffic. For those interested, my website is: http://ultimatebasictraining.com/admin/ (currently revaming my http://ultimatebasictraining.com website)
Intermediate & Advanced SEO | | StreetwiseReports0 -
Best solution to get mass URl's out the SE's index
Hi, I've got an issue where our web developers have made a mistake on our website by messing up some URL's . Because our site works dynamically IE the URL's generated on a page are relevant to the current URL it ment the problem URL linked out to more problem URL's - effectively replicating an entire website directory under problem URL's - this has caused tens of thousands of URL's in SE's indexes which shouldn't be there. So say for example the problem URL's are like www.mysite.com/incorrect-directory/folder1/page1/ It seems I can correct this by doing the following: 1/. Use Robots.txt to disallow access to /incorrect-directory/* 2/. 301 the urls like this:
Intermediate & Advanced SEO | | James77
www.mysite.com/incorrect-directory/folder1/page1/
301 to:
www.mysite.com/correct-directory/folder1/page1/ 3/. 301 URL's to the root correct directory like this:
www.mysite.com/incorrect-directory/folder1/page1/
www.mysite.com/incorrect-directory/folder1/page2/
www.mysite.com/incorrect-directory/folder2/ 301 to:
www.mysite.com/correct-directory/ Which method do you think is the best solution? - I doubt there is any link juice benifit from 301'ing URL's as there shouldn't be any external links pointing to the wrong URL's.0