Why might Google be crawling via old sitemap, when the new one has been submitted and verified?
-
We have recently relaunched Scoutzie.com and re-submitted our new sitemap to Google. When I look on Webmaster tools, our new sitemap has been submitted just fine, but at the same time, Google is finding a lot of 404s when crawling the site. My understanding, it is still using crawling the old links, which do not exists. How can I tell Google to refresh it's index and to stop looking at all the old links?
-
Yes it should. However, as Alan mentioned below, if you still have links pointing to the 404 pages, Google will always attempt to crawl them, and will keep you informed that you have errors.
If you do have external links to those 404 pages, you can 301 redirect them to an appropriate page using .htaccess. This way you'll keep the link value and also get rid of the Webmaster Tools error.
If you don't have any links to them, then yes, Google will eventually stop trying to crawl them.
-
It's very likely that we do. Given that I cannot track down a 1000+ links that now 404, will they eventually fall out by themselves, or do I have to tell Google that everything that's 404'ed should be dropped from crawl index? Thanks!
-
What if I simply pushed the new sitemap over the old one? In other words, scoutzie.com/sitemap is the same link, except now it contains the new map. That should be okay, right?
-
you may still have links pointing to those 404 pages on your site or externally. If not then eventually they will fall out of the index
-
Hey scoutzie,
This is actually covered pretty well in Joe Robison's blog post on fixing Webmaster Tools crawl errors: http://moz.com/blog/how-to-fix-crawl-errors-in-google-webmaster-tools
I'll quote the related info:
"One frustrating thing that Google does is it will continually crawl old sitemaps that you have since deleted to check that the sitemap and URLs are in fact dead. If you have an old sitemap that you have removed from Webmaster Tools, and you don’t want being crawled, make sure you let that sitemap 404 and that you are not redirecting the sitemap to your current sitemap."
Hope this helps, good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is a new website the only way to get better rankings?
Our website is https://www.franklin-bell.com.au. For a while now we've been trying to get a few of our pages on there to get a better rank on google. A local SEO agency told us that some of our pages are not like "the modern standard" for web pages, but our page score for the keywords we want is 90% and higher. Do we really need to completely rebuild the website (it was built in 2014) in order to get better rankings? Or is it sufficient to have the 97% page score and then just get some back links and social media referrals etc. Thanks in advance.
Moz Pro | | franklin-bell0 -
Site Crawl 4xx Errors?
Hello! When I check our website's critical crawler issues with Moz Site Crawler, I'm seeing over 1000 pages with a 4xx error. All of the pages that are showing to have a 4xx error appear to be the brand and product pages we have on our website, but with /URL at the end of each permalink. For example, we have a page on our site for a brand called Davinci. The URL is https://kannakart.com/davinci/. In the site crawler, I'm seeing the 4xx for this URL: https://kannakart.com/davinci/URL. Could this be a plugin on our site that is generating these URLs? If they're going to be an issue, I'd like to remove them. However, I'm not sure exactly where to begin. Thanks in advance for the help, -Andrew
Moz Pro | | mostcg0 -
Tracking Rankings In Google
Is there a way to see historical rankings of your site in Google. I have a site that has been running for a few years now. I want to see how are Google Ranking has performed throughout the years. We recently redesigned and suffered in rankings but I only know(from Google Analytics) What our traffic used to be, not what our rankings were .
Moz Pro | | pinksgreens0 -
Still Cant Crawl My Site
I've removed all blocks but two from our htaccess. They are for amazonaws.com to block amazon from crawling us. I did a fetch as google in our WM tools on our robots txt with success. SEOMoz crawler here hit's our site and gets a 403. I've looks in our blocked request logs and amazon is the only one in there. What is going on here?
Moz Pro | | martJ0 -
Are the SEOMoz queries banned by Google?
Hi: In our company, we have notified that Google is banning our IP due to massive queries under the same IP. It is caused mainly by local softwares installed in PCs or some plugins of Firefox. We think that SEOmoz don´t causes this due to its tool uses its own IP and servers, not ours. We would like your opinion about this. Thank you,
Moz Pro | | Equipo_SEO_HUB_Interaction0 -
Does Google Reward Bad Links in Some Industries?
This week I've been diving into competitive research again for several clients in different industries, and unless the data in Open Site Explorer is wrong, it really does appear that Google continues to reward sites with unnatural links in certain industries, especially industries that are less tech-savvy (agriculture, finance, manual labor). I've combed through all the links carefully, to ensure there aren't some secret high-quality links lurking, and for some of these sites that are showing up in the top ten for search they do not even one real, high-quality link. Also, their onsite optimization is often way over the top and the user experience is nothing less than nauseating. Could it be a MozScape issue, or does Google continue to reward terrible sites and link structures in certain industries? I'd love to hear your thoughts.
Moz Pro | | newwhy0 -
Order of urls in SEOMoz crawl report
Is there any rhyme or reason to the order of urls in the SEOMoz crawl report, or are the urls just listed in random order?
Moz Pro | | LynnMarie0 -
Hello I am new to SEO. Does SEOMoz have a to do list or can you send me one.
I just need to know what steps I should take to improve my site. My website url is : http://nuscopemed.com Thanks
Moz Pro | | NinaGraham0