How Often is Site Crawled
-
Good morning-
I saw some errors in my first crawl and immediately removed the pages from my website. I then re-created my XML sitemap and uploaded to Google.
The question I have is will the site be crawled to recognize the changes in the next day or so? The pages were just placed on the site as test pages and never removed. The initial crawl that notified me it was done found the errors and were removed.
Thanks for your help.
Peter
-
Oops sorry I misunderstood - thought you meant Google, but at least Keri jumped in there and gave you the answer
-
SEOmoz will crawl the site once a week. You can go to research tools and select a crawl on demand of up to 3000 URLs from a domain if you'd like a crawl sooner.
-
Thanks Matt. What I meant to ask was how often will SEO Moz crawl the site?
-
There is no set time limit to how often a site is crawled by Google - different sites have different crawl rates depending on factors such as how popular they are. For instance a well ranked, popular news site with content that is regularly being updated and added will have a crawl rate that is higher than a small personal site.
Have a look at this article in regards to ways you can encourage Google to crawl your site more often - http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/ (an old article but still applies)
Have you thought about using the URL removal tool in Google Webmaster Tools to quickly remove the test URLs - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=164734
And finally in relation to speeding up the crawling of specific pages on your site you can use the fetch as Googlebot tool > enter a URL > click submit URL to index to speed the crawling of that content up. Here is a great article on this much underused tool - http://searchenginewatch.com/article/2198245/How-to-Force-Google-to-Recrawl-Your-Site
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We are migrating a site and are seeing alot of 301s and 302s already in the old site is it ok to leave those as is?
For the 3xx’s I’m not sure if it’s okay for us to redirect to these so please advise on that
Technical SEO | | lina_digital0 -
Changing a site from http to https
Will my rankings be affected if I change domain from http to https and force redirect?
Technical SEO | | Clickatell20 -
Site offline - Mitigating measures?
Hi, Our domain has expired, and it could take up to 48h to recover our website. Appart from the obvious image damage, It worries me Google will just think we have vanisheg Any recommendations? Maybe update something on WebMasterTools? Not having the domain, cannot even do any temporary redirect, etc... Thanks! Jaime
Technical SEO | | BaseKit0 -
How is this site doing this?
http://www.meccabingo.com It shows a splash / promotion page yet you check the cache and it's the real homepage, they are doing this so they don't lose rankings but how are they redirecting users to that but Google is caching the real homepage? is it friendly? thanks!!
Technical SEO | | AdiRste0 -
Sitemap Creation + Site speed
Hi there, I am looking for a sitemap creation tool, so I can submit my site to Google. My site is www.vallnord.com On the other hadn I would like to speed up my web. Any tip? Regards, Guido.
Technical SEO | | SilbertAd0 -
Is it a good idea to make 301 from a site which you know google has banned certain keywords for to a new site with similar content
Here is a short question re. 301. I read Dovers article on how to move an old domain to a new one. Say you have been a little inexperienced regarding linkbuilding and used some cheap service in the past and you have steadily seen that certain keywords have been depreciating in the SERP - however the PR is still 3 for the domain - now the qustion is should you rediect with a 301 in .htaccess to a new domain when you know that google does not like certain keywords with respect to the old site. Will the doom and gloom carry over to the new site?
Technical SEO | | Kofoed0 -
Site problem
I moved a site earlier on in the year to a better server www.keyrs.co.uk, my main keywords being equity release - equity release calculator and equity release schemes. Since this happened the ranking have gone down and the schemes and calculator terms and have hit positions 7-8 when they were 2-3. basically my question is open to all, i am looking to see what the problem is with these pages as it is driving me nuts. All tools on SEO moz show the pages are doing well, however i must be missing something. Mike
Technical SEO | | TomBarker820