Why is Google Webmaster Tools reporting a massive increase in 404s?
-
Several weeks back, we launched a new website, replacing a legacy system moving it to a new server. With the site transition, webroke some of the old URLs, but it didn't seem to be too much concern. We blocked ones I knew should be blocked in robots.txt, 301 redirected as much duplicate data and used canonical tags as far as I could (which is still an ongoing process), and simply returned 404 for any others that should have never really been there.
For the last months, I've been monitoring the 404s Google reports in Web Master Tootls (WMT) and while we had a few hundred due to the gradual removal duplicate data, I wasn't too concerned. I've been generating updated sitemaps for Google multiple times a week with any updated URLs. Then WMT started to report a massive increase in 404s, somewhere around 25,000 404s per day (making it impossible for me to keep up). The sitemap.xml has new URL only but it seems that Google still uses the old sitemap from before the launch. The reported sources of 404s (in WMT) don't exist anylonger. They all are coming from the old site.
I attached a screenshot showing the drastic increase in 404s. What could possibly cause this problem?
-
Thank you for both responses...
Nakul--
I have been following everything exactly as you have described. In general the goal during the development was to keep changes to an absolute minimum. This has not always been possible.
The majority of external links have been 301 redirected or in cases where the new server responds to two differnet URLs for the same content a canonical tag has been added.
I have noticed that 99% of the reported URLs are former internal links. The reported 404s are completely out of proportion (194k vs less than 5k pages in the new xml sitemap).
I am really worried. Is there anything else I can do beside monitoring and hopping?
How long does it typically take to for "Things have to work their way out of its system."?
Is it possible that Google is somehow accessing the old IP address (although the DNS records for the domain have changed)? We left the old server alive and planning to shut it down after the second site has been moved away from it.
Thanks,
Adam
-
Agreed; it could an after effect and stems from inbound URLs to your site from other sites. That's what the majority of the 404s I see in GWT come from (vs being bad pages within my site).
Google probably isn't using the old sitemap if you gave them a new one. What could be happening is that it still needs to "reorganize" and reconcile your old URLs and new URLs. The indexed pages don't just disappear overnight or get replaced immediately because of a site map change. Things have to work their way out of its system.
If there's specific URLs you want to try to remedy immediately, look into the GWT Remove URL option under the optimization section.
-
What I'd suggest doing is randomly revising some of those 404's that appear and check whether they should indeed be 404s. Are there any bulk rules / wildcard 301s you can implement to redirect the traffic for 3-6 months ?
These URLs are usually found from external links to your website. When you click on a detail of any of the reported 404s, it tells you what the error details are, whether this link is in the sitemap or where it is linked from. You'd realize in most cases it's linked from somewhere. If it's an internal link, correct it. If it's external, do you think the webmaster might update it if you contact them or is it easier to just set a 301, retaining the SEO value ?
I hope this helps.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Homepage is deindexed in Google
Please help for some reason my website home page has disappeared, we have been working on the site but nothing that I can think of which would block it. There are no warnings in google console? Can anyone lend a hand in understanding what has gone wrong, I would really appreciate it. The site is: http://www.discountstickerprinting.co.uk/ Seems to be working again but I had to fetch the home page in google console, any idea why this has happened cannot afford a heat op at this age lol?
Intermediate & Advanced SEO | | BobAnderson0 -
Fetch as Google
I have odd scenario I don't know if anyone can help? I've done some serious speed optimisation on a website, amongst other things CDN and caching. However when I do a Search Console Fetch As Google It is still showing 1.7 seconds download time even though the cached content seems to be delivered in less than 200 ms. The site is using SSL which obviously creams off a bit of speed, but I still don't understand the huge discrepancy. Could it be that Google somehow is forcing the server to deliver fresh content despite settings to deliver cache? Thanks in advance
Intermediate & Advanced SEO | | seoman100 -
How to increase traffic?
This may seem a bit of a broad question but grateful for all knowledgeable input! There are many things to focus on when looking at organic SEO, but where should I be spending the most time/effort in order to increase traffic.
Intermediate & Advanced SEO | | seoman100 -
Google Rankings Are Dancing What To Do?
I was been ranking on page no.3 for many of my keywords. Suddenly they all dropped to Page No.9& Page no.10. Now 10 days back one of my keyword got back to page no.4 & now suddenly today it went back to page no.9 again. I haven't done any changes to the site. My page speed is less than 4secs. Checked everything & all seems fine. No errors or manual actions notification in Webmaster too. Should i wait for the rankings dance to get over or should i start doing something. Currently i am planning to get fresh pages with good content added to the website for the whole month of July. Suggest me what to do?
Intermediate & Advanced SEO | | welcomecure0 -
Webmaster Tools Internal Links
Hi all, I have around 400 links in the navigation menu (site-wide) and when I use webmaster tools to check for internal links to each page; some have as many as 250K and some as little as 200. Shouldn't the number of internal links for pages found in the navigation menu be relatively the same? Or is Google registering more internal links for pages linked closer to the top of the code Thanks!
Intermediate & Advanced SEO | | Carlos-R0 -
Can links indexed by google "link:" be bad? or this is like a good example by google
Can links indexed by google "link:" be bad? Or this is like a good example shown by google. We are cleaning our links from Penguin and dont know what to do with these ones. Some of them does not look quality.
Intermediate & Advanced SEO | | bele0 -
Ranking History Reports
I like that every week I can go into my campaign and see how I did. If I want to keep tabs (reports) every week and continuous record keeping how would I do that? For example, I want to see how we did last month on a particular keyword, should I set up to run a report every week for that keyword and have it emailed to me. Is that the only way to do that or does Moz keep previous history somewhere else? Thanks. PS Another website I help out on recently had a huge jump in pageviews this month. I don't track them in SEOMOZ currently is there anyway to figure out where that traffic is coming from? I am guessing perhaps they moved up in Google. Is there a way to see previous history? i.e. they are 33 last month for a certain keyword?
Intermediate & Advanced SEO | | greenhornet770 -
Google indexing flash content
Hi Would googles indexing of flash content count towards page content? for example I have over 7000 flash files, with 1 unique flash file per page followed by a short 2 paragraph snippet, would google count the flash as content towards the overall page? Because at the moment I've x-tagged the roberts with noindex, nofollow and no archive to prevent them from appearing in the search engines. I'm just wondering if the google bot visits and accesses the flash file it'll get the x-tag noindex, nofollow and then stop processing. I think this may be why the panda update also had an effect. thanks
Intermediate & Advanced SEO | | Flapjack0