Why might Google be crawling via old sitemap, when the new one has been submitted and verified?
-
We have recently relaunched Scoutzie.com and re-submitted our new sitemap to Google. When I look on Webmaster tools, our new sitemap has been submitted just fine, but at the same time, Google is finding a lot of 404s when crawling the site. My understanding, it is still using crawling the old links, which do not exists. How can I tell Google to refresh it's index and to stop looking at all the old links?
-
Yes it should. However, as Alan mentioned below, if you still have links pointing to the 404 pages, Google will always attempt to crawl them, and will keep you informed that you have errors.
If you do have external links to those 404 pages, you can 301 redirect them to an appropriate page using .htaccess. This way you'll keep the link value and also get rid of the Webmaster Tools error.
If you don't have any links to them, then yes, Google will eventually stop trying to crawl them.
-
It's very likely that we do. Given that I cannot track down a 1000+ links that now 404, will they eventually fall out by themselves, or do I have to tell Google that everything that's 404'ed should be dropped from crawl index? Thanks!
-
What if I simply pushed the new sitemap over the old one? In other words, scoutzie.com/sitemap is the same link, except now it contains the new map. That should be okay, right?
-
you may still have links pointing to those 404 pages on your site or externally. If not then eventually they will fall out of the index
-
Hey scoutzie,
This is actually covered pretty well in Joe Robison's blog post on fixing Webmaster Tools crawl errors: http://moz.com/blog/how-to-fix-crawl-errors-in-google-webmaster-tools
I'll quote the related info:
"One frustrating thing that Google does is it will continually crawl old sitemaps that you have since deleted to check that the sitemap and URLs are in fact dead. If you have an old sitemap that you have removed from Webmaster Tools, and you don’t want being crawled, make sure you let that sitemap 404 and that you are not redirecting the sitemap to your current sitemap."
Hope this helps, good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google RankBrain Update Affecting Ranking Reports?
In the past 2 months we have been seeing a large amount of movement in our ranking reports in Moz Pro. Here is what is going on: We run reports for around 15 different clients using Moz. We also track around an average of 100-150 different keywords for each client (some more, some less). What I have seen lately is that we have a week where the sites do great, next week a large shift down. The week after a large shift up. The next week a large shift down. Rinse, repeat. It makes it very hard to spot the trouble areas on a page or keyword when there doesn't really seem to be a consistent way of knowing what the trends are. It appears to be random, and the next week they seem to recover, along with some that do not. We will see some keywords drop 30 places, only to regain the majority of that drop the following week. Some of the sites have received content updates while others have remained largely untouched. It is very confusing to see this pattern, and I have tracked it for about 2 months. Is anyone seeing a similar pattern in their reports? Could this be related to the rollout of Google's RankBrain, the new AI learning tech from Google? Most of the keywords are tracked are not long tail, and are basic local business/service type of keywords. Plumber in St Louis, etc etc
Moz Pro | | David-Kley1 -
Crawl Diagnostics: Next crawl date is in the past
Hi - I have quite a few crawl diagnostic errors and warnings. I have attempted to fix many of them but noticed this note at the bottom of the crawl diagnostics chart: "Last Crawl Completed: Mar. 22nd, 2013 Next Crawl Starts: Mar. 29th, 2013" It looks like SEOMoz thinks the next crawl date is Mar 29th, 2013, which is two weeks ago. Is there any way to "force" the crawl and get it back on regular schedule? This may have happened when my account was disabled because my credit card expired...Thoughts?
Moz Pro | | 6thirty0 -
SEOMOZ Crawling Our Site
Hi there, We get a report from SEOMOZ every week which shows our performance within search. I noticed for our website www.unifor.com.au that it looks through over 10,000 pages, however our website sells less than 500 products so not sure why or how so many pages are trawled? If someone could let me know that would be great. It uses up a lot of bandwidth doing each of these searches so if the amount of pages being trawled reduced it would definitely assist. Thanks, Geoff
Moz Pro | | BeerCartel750 -
Exporting Google and Bing Search Results
Is there away to get a spreadsheet of the pages indexed for a certain domain in google and bing? i.e. I search google for Site:www.domain.com and I want to export a .csv file of all those domains/pages. Cheers
Moz Pro | | JohnW-UK0 -
Can we add sites to the crawl queue for OSE?
Is it possible to request that Open Site Explorer crawls a new URL on its next run? This tool is the first place I go to when working on a new site, and when there is "No Data Available" this is a little frustrating. I fully appreciate that this lack of data is usually a signal that the website is either very new or of low quality, however that if often the reason that I am brought in and would very much like to benchmark and provide initial analysis using this tool. It would make sense that OSE crawls the sites that Moz members are working on wouldnt it? Scott.
Moz Pro | | eseyo0 -
Does Google have a direct link with facebook and twitter?
Google monitor social media. What I'm wondering is do Google use the same tools we have on Facebook's API, Twitter's API etc to use in their SERPs Or do Facebook grant Google more detailed access to see who has liked links etc. I think it's quite an interesting point as surely I can push up my own count by repeatedly sharing my own links, which wouldn't be genuine. If Google had better access they could then determine what's been faked etc.
Moz Pro | | PhotoGazza0 -
Google analytics sync with SEOMOZ
Hey guys, I'm looking to sync our clients google analytic profiles to our clients campaigns in SEO MOZ. However all our clients google analytic profiles are on the one google account and not separate accounts. So I'm asking does SEO MOZ only allow you to grab the traffic data of one clients profile from a google account instead of multiple profiles from the same google account for different campaigns? If so has anybody got a workaround or any suggestions? If I haven't explained myself clearly let me know and I'll break it down further. Thanks again. Rob
Moz Pro | | daracreative0 -
Can I change the crawl day ?
Hi All I hope there is a simple solution to this - we have a number of campaigns setup which are all crawled, and therefore updated, on different days of the week. We review these weekly and it would be much easier if they were all crawled on the same day. Is it possible to change the crawl day for some campaigns? Thanks Roy
Moz Pro | | bluelogic0