2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
-
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages.
However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following.
2/1 - 4,373 pages crawled
2/2 - 367 pages crawled
2/3 - 4,777 pages crawled
2/4 - 437 pages crawled
What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day.
Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site.
Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day.
Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP
The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
-
2 weeks later, 2 more large spikes of 10X average...nothing bad is happening...oh well...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which page speed tools do you trust?
We are at a loss. We have been following Google Pagespeed Insights which gives us horrible marks. Nothing we do seems to really make a difference. GTMetrix and Pingdom give us much better scores. We don't know what to do...
Reporting & Analytics | | HashtagHustler0 -
Google Analytics reporting traffic for 404 pages
Hi guys, Unique issue with google analytics reporting for one of our sites. GA is reporting sessions for 404 pages (landing pages, organic traffic) e.g. for this page: http://www.milkandlove.com.au/breastfeeding-dresses/index.php the page is currently a 404 page but GA (see screenshot) is reporting organic traffic (to the landing page). Does anyone know any reasons why this is happening? Cheers. http://www.milkandlove.com.au/breastfeeding-dresses/index.php GK0zDzj.jpg
Reporting & Analytics | | jayoliverwright2 -
Google Crawl Stats
Hi all Wondering if anyone could help me out here. I am seeing massive variations in WMT of google crawl stats on a site I run. Just wondering if this is normal (see attached image). The site is an eccommerce site and gets a handful of new products added every couple of weeks. The total no of products is about 220k so this is only a very small %. I notice in WMT I have an amber warning under Server connectivity. About 10 days back I had warnings under DNS, Server and Robots. This was due to bad server performance. I have since moved to a new server and the other two warnings have gone back to green. I expect the Server connectivity one to update any day now. Ive included the graph for this incase it is relevant here. Many thanks for assistance. Carl crawlstats.png connect.png
Reporting & Analytics | | daedriccarl0 -
Site account in Google Analytics
Hello I have a question about my site account. On 2014, during a week, my ID tracking of Google Analytics was removed of the site, in this period the volume of users and sessions is lower than the other weeks. But I don't understand why are the sessions and users still reporting during this period without ID Tracking
Reporting & Analytics | | Arkix0 -
Google Analytics Tracking Code Problems
I am having a problem with my Google Analytics tracking codes. The relevant GA tracking codes have been entered into the source code of certain sites I'm running campaigns on, but Google Analytics is still telling me that the tracking code hasn't been installed - yet it has. What could possibly be the problem here? I have checked that these sites don't run off multiple domains, and that the tracking code has been entered in the correct place within the source code - and yet I am still having problems with GA. Any suggestions?
Reporting & Analytics | | WebhouseISP0 -
How do I set a filter in Google Analytics?
I want to filter out direct traffic from the service provider msn. For some reason they are crawling our traffic and its throwing off my data. So I want to exclude direct, but only from msn. Would I set my first filter for direct then msn or vice versa? Thanks
Reporting & Analytics | | EcommerceSite0 -
Google Analytics question
Hi, I recently relaunched my site on a wordpress CMS. The relaunch has gone well except for a category of traffic within Google Analytics which is referral traffic from google.co.uk (its a UK site) which has dropped off hugely. The non-paid search traffic from google.co.uk is the same or better than it was. Can anyone tell me what the difference is between referral traffic from google.co.uk and non-paid search traffic from google.co.uk. Thanks for your help. Ben
Reporting & Analytics | | benacuity0 -
Using Regex for Goals in Google Analytics
I’ve got a website with 20 different forms; and I’d like to track all 20 form completions with one Goal in Google Analytics. The completion-page URLs follow the pattern below (the website uses Drupal CMS) Form One: www.acme.com/node/2990/done?sid=651 Form Two: www.acme.com /node/2991/done?sid=785 Form Three: www.acme.com /node/2992/done?sid=1021 Form Four: www.acme.com /node/2993/done?sid=459 I believe that there is a way I can use regex so Google Analytics will track if any of these completion URLs appears. Looking for guidance how to format the regex statement.
Reporting & Analytics | | TopFloor0