2 days in the past week Google has crawled 10x the average pages crawled per day. What does this mean?
-
For the past 3 months my site www.dlawlesshardware.com has had an average of about 400 pages crawled per day by google. We have just over 6,000 indexed pages.
However, twice in the last week, Google crawled an enormous percentage of my site. After averaging 400 pages crawled for the last 3 months, the last 4 days of crawl stats say the following.
2/1 - 4,373 pages crawled
2/2 - 367 pages crawled
2/3 - 4,777 pages crawled
2/4 - 437 pages crawled
What is the deal with these enormous spike in pages crawled per day? Of course, there are also corresponding spikes in kilobytes downloaded per day.
Essentially, Google averages crawling about 6% of my site a day. But twice in the last week, Google decided to crawl just under 80% of my site.
Has this happened to anyone else? Any ideas? I have literally no idea what this means and I haven't found anyone else with the same problem. Only people complaining about massive DROPS in pages crawled per day.
Here is a screenshot from Webmaster Tools: http://imgur.com/kpnQ8EP
The drop in time spent downloading a page corresponded exactly to an improvement in our CSS. So that probably doesn't need to be considered, although I'm up for any theories from anyone about anything.
-
2 weeks later, 2 more large spikes of 10X average...nothing bad is happening...oh well...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Fire a tag when element is loaded on page (Google Tag Manager)
I'm using an Element Visibility trigger to track a value that appears on a page. However, I want to track this value even when the user doesn't scroll to the area of the page where the element is (i.e. when the page is loaded, and the value is displayed below the fold, but the user doesn't scroll down there). Is there a way of doing this
Reporting & Analytics | | RWesley0 -
Understanding Average Position in Google Anaylitics
Hello here, I have a question about the Queries report under "Search Engine Optimization" in Google Analytics: is the "Average Position" information a reliable one? I have a lot of queries that appear, from that report, to average first position, but when I verify that on Google by connecting anonymously, I can't even find my result on the first page! To me, that information is worthless and makes me think all the rest of that report is unreliable. If anyone can help me to understand it, I'd really appreciate it. Thank you in advance for any thoughts.
Reporting & Analytics | | fablau0 -
Google is not indexing all URLs
My website have company and events profile from 200 countries. So it does have lots of URL. Earlier in August 2014, Google used to crawl 90% of URLs we submit. Thing goes wrong when we shifted from http to https. We lost traffic. But we are gaining it slowly. Main concern is that, It still does not indexed all submitted URLs. It have crawled merely 8% of all URLs submitted. site address is businessvibes.com Any help would be appreciated.
Reporting & Analytics | | irteam0 -
Exact Match in Google Search (Not Adwords)
I was going throught the list of keywords that have sent traffic to my site over the last 7 years and cam across one "A516 grade 70" that had hundreds of variants. Now in a lot of cases search volumes were different as were SERPS. We've tested a few variants with reworked pages (70% similar to original but optimised for variant keyword) and see good SERPS and traffic results. Theres obviously some diminishing returns here for us but the interesting question is when to these variants become an exact match and when not? In some cases the variants are unique because of the spacig, periods and hyphens used. there isn't a clear correlation with exact matc though. Insight appreciated. (Sorry for spelling errors. Form doesn't play nicely with iPad)
Reporting & Analytics | | Zippy-Bungle0 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
Google Analytice for Shopping Results
We are trying to optimise for Google Shopping and therefore would like to be able to monitor traffic to site from this source? How can this be done in Google Analytics? Thanks
Reporting & Analytics | | seanmccauley0 -
Why isn´t my google analytics registering goals?
Hi everyone! I know this is is not the standard SEO question, but I hope you can help me with your knowledge. I set some goals in GA for the thankyou page that appears when a client requests more information on a property for sale. Problem is I have been testing them and the goals don´t get registered. I don´t know what am I doing wrong and I would really appreciate if you could help me through some light on this. I have attached an image of the option for the goal in GA. Can anybody help me? Paroa.png
Reporting & Analytics | | inmonova0 -
Why does Google Analytics think PPC traffic is organic?
I have a bastard of a problem... Google Analytics is incorrectly tracking PPC traffic as SEO which is screwing up all my reporting . I don't care for rankings, I care for actual SEO traffic and I can't be sure that what i am seeing is correct which is driving me nuts. Any ideas?
Reporting & Analytics | | Red_Mud_Rookie1