How Often is Site Crawled
-
Good morning-
I saw some errors in my first crawl and immediately removed the pages from my website. I then re-created my XML sitemap and uploaded to Google.
The question I have is will the site be crawled to recognize the changes in the next day or so? The pages were just placed on the site as test pages and never removed. The initial crawl that notified me it was done found the errors and were removed.
Thanks for your help.
Peter
-
Oops sorry I misunderstood - thought you meant Google, but at least Keri jumped in there and gave you the answer
-
SEOmoz will crawl the site once a week. You can go to research tools and select a crawl on demand of up to 3000 URLs from a domain if you'd like a crawl sooner.
-
Thanks Matt. What I meant to ask was how often will SEO Moz crawl the site?
-
There is no set time limit to how often a site is crawled by Google - different sites have different crawl rates depending on factors such as how popular they are. For instance a well ranked, popular news site with content that is regularly being updated and added will have a crawl rate that is higher than a small personal site.
Have a look at this article in regards to ways you can encourage Google to crawl your site more often - http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/ (an old article but still applies)
Have you thought about using the URL removal tool in Google Webmaster Tools to quickly remove the test URLs - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=164734
And finally in relation to speeding up the crawling of specific pages on your site you can use the fetch as Googlebot tool > enter a URL > click submit URL to index to speed the crawling of that content up. Here is a great article on this much underused tool - http://searchenginewatch.com/article/2198245/How-to-Force-Google-to-Recrawl-Your-Site
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Query on Site Architecture
Hi All, When I check on my ecommerce site in one of the architecture tool in that my Ecommerce Homepage interlink with 765 pages whereas when I check few competitors and big brands then there homepage linked with 28 pages, 33, 47, 57 etc not like my site 765 pages. Do I am wrong anywhere? Can you please check the screenshot of mine & one of the competitor's site architecture? Because as per me site architecture also play good role in google organic ranking. vXs5dh2 16wre
Technical SEO | | pragnesh96390 -
Then why my site is not ranking
My website's DA and PAs are good compare with my competitors. Then why my site is not ranking.
Technical SEO | | Somanathan0 -
Site dropped after article submissions
I have a site about back pain. The name is lower back pain relief. It has a lot of content on how to treat back pain with natural remedies. I paid for botw.org and yahoo directories as well as submissions from submit edge. My site was doing well within a few months. I was floating around at the bottom of page two for "lower back pain relief" and lower back pain. I found out about Unique article wizard. I started using that to write and submit articles. Now, my website is not ranking anywhere. I submitted a request for reconsideration to Google. They said no manual action had been taken. The question: How long does this penalty last? Is that website toast now? I think it was penalized for using the anchor text too much or adding too many links at once. It has been penalized for about 6 weeks now.
Technical SEO | | Naturalhealth150 -
SEOMoz Crawling Errors
I recently implemented a blog using WordPress on our website. I didn't use WordPress as the CMS for the rest of our site just the blog portion. So as an example I installed Wordpress in http://www.mysite/blog/" not in the root. My error report in SEOMoz went from 0 to 22e. The Moz bot or crawler that SEOMoz uses is reporting a ton of 4xx errors to strang links that shouldn't exist anywhere on the site. Example: Good link - http://www.mysite/products.html Bad link reported by SEOMoz - http://www.mysite/blog/my-first-post/products.html I've also noticed that my page speed as become much slower as reported by Google. Does anybody know what could be happening here? I know that typically it's better to install WordPress in the root and use it to control the entire site but I was under the gun to get a blog out. Thanks
Technical SEO | | TRICORSystems0 -
Site maps, Is there any benefit?
I have a relatively simple and small site (60 pages). All of it is crawlable and there is nothing I want non follow. So, is there any real benefit to a sitemap since Google can get to all the site anyway? Do they give the site more credence or something because it's there? I guess as an aside, are there any favorite sites that will generate a site map? Thanks!
Technical SEO | | Banknotes0 -
What keywords should i be using to promote my site
Hi i am looking to promote my home page which is a lifestyle magazine www.in2town.co.uk and i am not sure what keywords i should be using to promote it. I am doing ok for the keyword lifestyle magazine but i am struggling on what other keywords i should be using to get people to the home page of the magazine. The magazine is nearly finished and we still have a couple of finishing touches to do but the basics of the magazine is as follows holiday and travel news, soap gossip, celebrity gossip, product reviews, lingerie brands, gastric band hypnotherapy, health, fashion and beauty and holiday reviews. I want the home page to be the main page where everyone visits but i am not sure what i should be doing to accomplish this. Any ideas would be of a great help
Technical SEO | | ClaireH-1848860 -
Site Structure question
when deciding the Site structure for a e-commerce site Is it better to keep everything mysite.com/widget.html or use categories like mysite.com/Gifts/widget.html
Technical SEO | | DavidKonigsberg0 -
Problem with my site
the site is casino.pt we created the site 7-8 month ago, we started to push it by good and natural links (http://www.opensiteexplorer.org/www.casino.pt/a!links!!filter!all!!source!external!!target!page), links in sites with content rich and most of them related to gambling and sport topics. During the first 3-5 months, the rankings were better and better, after the 6 months, the site lose all its rankings. Aditional details http://www.casino.pt/robots.txt http://www.google.pt/#hl=pt-PT&source=hp&biw=1280&bih=805&q=site:http%3A%2F%2Fwww.casino.pt&aq=f&aqi=&aql=&oq=&fp=2651649a33cd228 no critical errors in google webmaster tools any idea how can I fix it? thanks
Technical SEO | | Yaron530