New CMS system - 100,000 old urls - use robots.txt to block?
-
Hello.
My website has recently switched to a new CMS system.
Over the last 10 years or so, we've used 3 different CMS systems on our current domain. As expected, this has resulted in lots of urls.
Up until this most recent iteration, we were unable to 301 redirect or use any page-level indexation techniques like rel 'canonical'
Using SEOmoz's tools and GWMT, I've been able to locate and redirect all pertinent, page-rank bearing, "older" urls to their new counterparts..however, according to Google Webmaster tools 'Not Found' report, there are literally over 100,000 additional urls out there it's trying to find.
My question is, is there an advantage to using robots.txt to stop search engines from looking for some of these older directories? Currently, we allow everything - only using page level robots tags to disallow where necessary.
Thanks!
-
Great stuff..thanks again for your advice..much appreciated!
-
It can be really tough to gauge the impact - it depends on how suddenly the 404s popped up, how many you're seeing (webmaster tools, for Google and Bing, is probably the best place to check) and how that number compares to your overall index. In most cases, it's a temporary problem and the engines will sort it out and de-index the 404'ed pages.
I'd just make sure that all of these 404s are intentional and none are valuable pages or occurring because of issues with the new CMS itself. It's easy to overlook something when you're talking about 100K pages, and it could be more than just a big chunk of 404s.
-
Thanks for the advice! The previous website did have a robots.txt file with a few wild cards declared. A lot of the urls I'm seeing are NOT indexed anymore and haven't been for many years.
So, I think the 'stop the bleeding' method will work, and I'll just have to proceed with investigating and applying 301s as necessary.
Any idea what kind of an impact this is having on our rankings? I submitted a valid sitemap, crawl paths are good, and major 301s are in place. We've been hit particularly hard in Bing.
Thanks!
-
I've honestly had mixed luck with using Robots.txt to block pages that have already been indexed. It tends to be unreliable at a large scale (good for prevention, poor for cures). I endorsed @Optimize, though, because if Robots.txt is your only option, it can help "stop the bleeding". Sometimes, you use the best you have.
It's a bit trickier with 404s ("Not Found"). Technically, there's nothing wrong with having 404s (and it's a very valid signal for SEO), but if you create 100,000 all at once, that can sometimes give raise red flags with Google. Some kind of mass-removal may prevent problems from Google crawling thousands of not founds all at once.
If these pages are isolated in a folder, then you can use Google Webmaster Tools to remove the entire folder (after you block it). This is MUCH faster than Robots.txt alone, but you need to make sure everything in the folder can be dumped out of the index.
-
Absolutely. Not founds and no content are a concern. This will help your ranking....
-
Thanks a lot! I should have been a little more specific..but, my exact question would be, if I move the crawlers' attention away from these 'Not Found' pages, will that benefit the indexation of the now valid pages? Are the 'Not Found's' really a concern? Will this help my indexation and/or ranking?
Thanks!
-
Loaded question without knowing exactly what you are doing.....but let me offer this advice. Stop the bleeding with robots.txt. This is the easiest way to quickly resolve that many "not found".
Then you can slowly pick away at the issue and figure out if some of the "not founds" really have content and it is sending them to the wrong area....
On a recent project we had over 200,000 additional url's "not found". We stopped the bleeding and then slowly over the course of a month, spending a couple hours a week, found another 5,000 pages of content that we redirected correctly and removed the robots....
Good luck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Url permalink structure question!
Hello! I just read the "15 SEO Best Practices for Structuring URLs" but I have still a question: My A version bellow "20 accessoires" has no meaning in french. If I add "voyage" (version B), then is it considered as spammy? I mean the "voyage" keyword repetition? A) http://www.lytchee.com/preparer-son-voyage/20-accessoires/ B) http://www.lytchee.com/preparer-son-voyage/20-accessoires-voyage Thanks for advices! Sylvain
On-Page Optimization | | lytcheetv0 -
How can i block the below URLs
Google indexed plugins pages for my website. Please check below. How can stop them to be indexed on google.? http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/glass/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/borderlesslight3d/ http://www.ayurjeewan.com/wp-content/plugins/LayerSlider/static/skins/defaultskin/ My robots.txt file is - User-agent: * Disallow: /wp-admin/
On-Page Optimization | | MasonBaker0 -
Using a keyword on homepage of a blog
I have a blog and the homepage has the 5 most recent posts. I ran a report card on my homepage for my main keyword. One of the problems is that the keyword only appears 1 time. I don't want to put it in the signature of every post because I found that causing problems with self-cannibalizing. I checked my competitor and they got a check mark for this but I looked at their homepage and I found the keyword NOWHERE! So where is my competitor hiding the keywords and how can I get the keywords on the homepage when the content is constantly changing? Thanks in advance!
On-Page Optimization | | 2bloggers0 -
I have more pages in my site map being blocked by the robot file than I have being allowed to be crawled. Is Google going to hate me for this?
Using some rules to block all pages which start with "copy-of" on my website because people have a bad habit of duplicating new product listings to create our refurbished, surplus etc. listings for those products. To avoid Google seeing these as duplicate pages I've blocked them in the robot file, but of course they are still automatically generated in our sitemap. How bad is this?
On-Page Optimization | | absoauto0 -
Appropriate Use of Rel Canonical
I have encountered problems regarding rel canonical. When I ran On-Page Report Card it says **Error: ** Appropriate Use of Rel Canonical Canonical URL: "http://www.sourcedental.createmyid.net/teeth-whitening/" Explanation: If the canonical tag is pointing to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. Make sure you're targeting the right page (if this isn't it, you can reset the target above) and then change the canonical tag to reference that URL. Recommendation We check to make sure that IF you use canonical URL tags, it points to the right page. If the canonical tag points to a different URL, engines will not count this page as the reference resource and thus, it won't have an opportunity to rank. If you've not made this page the rel=canonical target, change the reference to this URL. NOTE: For pages not employing canonical URL tags, this factor does not apply." I just don't know how to fix this. I am using Wordpress SEO by Yoast but I haven't change any settings regarding rel canonical. Can anyone help me with this? Thanks
On-Page Optimization | | projectassistant0 -
Long Url but makes no sense
Hi Just joined. Crawl states that I am getting a lot of errors, looks like the spider is getting confused and looping back on itself ? Is there a way to see where the crawl was formulated (ie where from) ? It is generating urls like: http://www.wickman.net.au/wineauction/wine_auction_alert.aspx/auction/auction/auction/auction/auction/auction/Default.aspx from http://www.wickman.net.au/wineauction/wine_auction_alert.aspx
On-Page Optimization | | blinkybill0 -
Close URL owned by competitors.
The following example is exactly analogous to our situation (site names slightly altered😞 We own www.business-skills.com. It's our main site. We don't own, and would rather avoid paying for, www.businessskills.com. It's a parked domain and the owners want a very large sum for it. We own www.business-skills.co.uk and point it to our main site. We don't own www.businessskills.co.uk. This is owned by our biggest competitor. We also own www.[ourbrand].com and .co.uk, and point them to the main site. My question is - how much traffic do you think we may be missing due to these nearly-but-not-quite URL matches? Does it matter in terms of lost revenue? What sort of things should I be looking at to get a very rough estimate?
On-Page Optimization | | JacobFunnell0 -
Url question for seo
Would it be beneficial to use url that is a match for my keyword to help with seo, then have my currently url forward to that one so I don't have to change any marketing materials? I was one of the feedback that I got when doing the on page keyword optimization tool on seo moz. Thanks J
On-Page Optimization | | fertilityhealth0