Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's Causing My Extremely Low Bounce Rate
My client's site that is reporting an under 10% bounce rate for all sources. Direct is the highest at 8%. I'm no expert in GA but wondering if there is a problem with the analytics/tag manager code on the site. I'm especially concerned about the GTM body script being in an iframe which I read could be trouble. <!-- Google Tag Manager (noscript) -->
Reporting & Analytics | | bradsimonis
<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-MWGMNW6"
height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) --> You can see all the source code here:
view-source:https://nfinit.com/0 -
My website has been declining in rankings!
Hello everyone, I have been doing the SEO on the following site: www.painters-decorators-london.co.uk for the past few years and the rankings are slowly declining since last year. (not the recent Google update). I can't seem to figure out what exactly is causing this to happen The page authority and domain authority is higher than my competitors. The on page SEO I believe is done appropriately but I'm not sure whats causing this decline. How can I find out exactly using the MOZ Pro account that I have. I don't think I'm totally aware of the important metrics that one can use on this site to determine the problem. Please advise!
Reporting & Analytics | | Ready2Paint0 -
Tracking time spent on a section of a website in Google Analytics
Hi, I've been asked by a client to track time spent or number of pages visited on a specific section of their website using Google Analytics but can't see how to do this. For example, they have a "golf" section within their site and want to measure how many people either visit 5 page or more within the golf section or spend at least 6 minutes browsing the various golf section pages. Can anyone advise how if this can be done, and if so, how I go about it. Thanks
Reporting & Analytics | | geckonm0 -
Bounce Rate Question - The percent calculated does not add up
Hello All, I'm attempting to see why organic search bounce rate has increased by 5% when compared to last year for a certain section of my website. I am using a custom segment to filter the specific pages I want to look at. Once the custom segment is set, I go to Acquisition - > Channels - > Organic. Then, I click the Landing Pages tab. Because we don't have keyword data anymore the only thing I can look at is the landing pages that contributed to the change in bounce. Finally, I set my date range and compare to the same date range as last year. Once I set the date range I am presented with a list of URLs and the percent change in bounce rate for each URL. This is where I get confused. If you look at the average bounce rate at the top of the column (example 1 attached) it does not add up with the data below it. If you export all of the data to excel, and then do an "Average" function in Excel, the data adds up to 17.29% instead of 35.04% for Sept. 2013. Why does this not add up? Isn't GA calculating the Average? Also, I always notice several URLs with only 1 session per URL. Several of these 1 session URLs have a 100% bounce rate. Since the bounce rate at the top of the column (example1) is a reflection of the average bounce rate, wouldn't these 1 session URLs significantly distort my data? I ultimately just want to see the pages that are contributing to the increased bounce rate when compared to last year. Having a hard time figuring this one out. Thank you all, Dave zMfAGls
Reporting & Analytics | | DaveGuyMan0 -
What penalty might have hit here (screenshot attached)
I've had a spiky organic traffic profile for some time and wondered if anyone could suggest what penalty might have hit me. Something did in July but then when it recovered (not sure how) I ended up with double the traffic. Then it dipped, recovered a but and has slowly declined since. I've had no manual penalties so it must be algorithmic which means I can fix it. The problem is I don't know what the issues is! Any help appreciated! Ae00Lzl.png
Reporting & Analytics | | SamCUK0 -
The client's website serves as the main referral?
Hi mozzers, I have this weird case where one of my client's first referral is its own website!! I am really confused especially that I have checked there www vs non www and the non www is redirected to the www. This means that it resolve to one version which is good! Any thoughts on why the main referral is its own site? Thanks
Reporting & Analytics | | Ideas-Money-Art0 -
If I am changing my domain for my website and want to keep using the same Google Analytics account to keep the data from the old domain. How should I proceed?
If I am changing my domain for my website and want to keep using the same Google Analytics account to keep the data from the old domain. How should I proceed? Do I have to start a new Google Analytics account for the new domain? If so how do I keep the old data? Or can I use the same GA account? Thank you.
Reporting & Analytics | | brianhughes1 -
Open explorer is not finding my website information yet?
Majestic SEO and even yahoo sitemap see's it... why not SEOMOZ? The website is a few months old already. www.lackofsleephq.com
Reporting & Analytics | | sleepmaster0