New CMS system - 100,000 old urls - use robots.txt to block?
-
Hello.
My website has recently switched to a new CMS system.
Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls.
Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical'
Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find.
My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary.
Thanks!
-
Great stuff..thanks again for your advice..much appreciated!
-
It can be really tough to gauge the impact - it depends on how suddenly the 404s popped up, how many you're seeing (webmaster tools, for Google and Bing, is probably the best place to check) and how that number compares to your overall index. In most cases, it's a temporary problem and the engines will sort it out and de-index the 404'ed pages.
I'd just make sure that all of these 404s are intentional and none are valuable pages or occurring because of issues with the new CMS itself. It's easy to overlook something when you're talking about 100K pages, and it could be more than just a big chunk of 404s.
-
Thanks for the advice! The previous website did have a robots.txt file with a few wild cards declared. A lot of the urls I'm seeing are NOT indexed anymore and haven't been for many years.
So, I think the 'stop the bleeding' method will work, and I'll just have to proceed with investigating and applying 301s as necessary.
Any idea what kind of an impact this is having on our rankings? I submitted a valid sitemap, crawl paths are good, and major 301s are in place. We've been hit particularly hard in Bing.
Thanks!
-
I've honestly had mixed luck with using Robots.txt to block pages that have already been indexed. It tends to be unreliable at a large scale (good for prevention, poor for cures). I endorsed @Optimize, though, because if Robots.txt is your only option, it can help "stop the bleeding". Sometimes, you use the best you have.
It's a bit trickier with 404s ("Not Found"). Technically, there's nothing wrong with having 404s (and it's a very valid signal for SEO), but if you create 100,000 all at once, that can sometimes give raise red flags with Google. Some kind of mass-removal may prevent problems from Google crawling thousands of not founds all at once.
If these pages are isolated in a folder, then you can use Google Webmaster Tools to remove the entire folder (after you block it). This is MUCH faster than Robots.txt alone, but you need to make sure everything in the folder can be dumped out of the index.
-
Absolutely. Not founds and no content are a concern. This will help your ranking....
-
Thanks a lot! I should have been a little more specific..but, my exact question would be, if I move the crawlers' attention away from these 'Not Found' pages, will that benefit the indexation of the now valid pages? Are the 'Not Found's' really a concern? Will this help my indexation and/or ranking?
Thanks!
-
Loaded question without knowing exactly what you are doing.....but let me offer this advice. Stop the bleeding with robots.txt. This is the easiest way to quickly resolve that many "not found".
Then you can slowly pick away at the issue and figure out if some of the "not founds" really have content and it is sending them to the wrong area....
On a recent project we had over 200,000 additional url's "not found". We stopped the bleeding and then slowly over the course of a month, spending a couple hours a week, found another 5,000 pages of content that we redirected correctly and removed the robots....
Good luck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is using hyphens in a URL to separate words good practice?
Hi guys, I have a client who wants to use a hyphen to separate two words in the URL to make each work stand out. Is is good or bad practice to use a hyphen in a URL and will it affect rankings? Thanks!
On-Page Optimization | | StoryScout0 -
Backlink URL: With or Without WWW?
When it comes to backlinks. Does it matter with or without WWW? For example my website is without WWW and I backlink with WWW, will it still affect my website rank?
On-Page Optimization | | Japracool0 -
Two URL's for the same page
Hi, on our site we have two separate URL's for a page that has the same content. So, for example - 'www.domain.co.uk/stuff' and 'www.domain.co.uk/things/stuff' both have the same content on the page. We currently rank high in search for 'www.domain.co.uk/things/stuff' for our targeted keyword, but there are numerous links on the site to www.domain.co.uk/stuff and also potentially inbound links to this page. Ideally we want just the www.domain.co.uk/things/stuff URL to be present on the site, what would be the best course of action to take? Would a simple Canonical tag from the '/stuff' URL which points to the '/things/stuff' page be wise? If we were to scrap the '/stuff' URL totally and redirect it to the 'things/stuff' URL and change all our on site links, would this be beneficial and not harm our current ranking for '/things/stuff'? We only want 1 URL for this page for numerous reasons (i.e, easier to track in Analytics), but I'm a bit cautious that changing the page that doesn't rank may have an affect on the page that does rank! Thanks.
On-Page Optimization | | Jaybeamer2 -
I am optimizing title tags and was wondering if it makes a difference if I use "commas" in between keywords that are synonyms or should I use "and" instead?
For example: "pants, trousers at pants.com" or "pants and trousers at pants.com".
On-Page Optimization | | EcomLkwd0 -
Getting Google to provide a different URL in SERP
For one of my client’s sites, I have several keywords that are ranking in the top 5 positions. However, they have a high bounce rate. I believe this is because Google is delivering a different URL than the page we have optimized for the keyword. Any suggestions on ways I can get Google to present our preferred page?
On-Page Optimization | | TopFloor0 -
Old pages
I have a site where I have 5,000 new products each year, I never waned to deleted the old pages due to links pointing to them and keywords. But I now have 20,000 plus pages, does having that many pages spread out my link juice or does it effect me in any other ways over having a site with 5,000 pages or should I keep not deleting old pages so I dont loose any links? Along with that I currently do not link to my old pages from my site so Im guessing google does not get to them very often if at all, if you agree to still keep them should I link to them somewhere? Because the products are not that simiiar and they do bring added value I dont think canonical would work here
On-Page Optimization | | Dirty0 -
Using new domain in existing website
We have already a site and we want to make business in another country. We don't want to make a copy of our site because it requires more resources, but we want to use a different domain because the business will be run by a partner in that country. The idea is to make a folder in our site, www.mysite.com/countryname/ and associate a new domain www.newdomain.com, so when users go to the new domain they see the content under www.mysite.com/country/. Since de www.newdomain.com will use DNS redirection, the current domain won't be seen. Is this correct from a SEO perspective? Thanks!
On-Page Optimization | | Xopie1 -
Keywords in URL:
what kind of URL should we use? www.keyword.net/keyword-city or www.keyword.net/city which URL you would prefer?
On-Page Optimization | | alibeef0