Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you Thomas.
-
Just to add to this, there is nothing inherently wrong with Google crawling more pages of your site. The only time I would modify the crawl rate is when the extra crawling is actually slowing your server down.
-
Hi There,
The crawl rate control was devised by Google to give control to the users, so that they can limit the server load that is created by constant crawling of the website.
So, it's up to you to decide whether you want to lower/limit it.
https://support.google.com/webmasters/answer/48620?hl=en
Thanks,
Vijay
-
Thank you Vijay, your response is very helpful. Do you know if there are any guidelines for optimal crawl rates? I tend to look at average pages crawled per day and multiply by 90. If that number is equal to or more than the amount of pages on-site, then we'd be good, right? Or is there a flaw in that logic?
-
Hi Thomas,
Thank you for responding. Yes, kind of. There are 40 main categories and each of those has upto 100 links to sub categories, and then the same again for sub-sub categories.
I've spent the last year cleaning it up and removing pages that didnt need to be there. Quite a lot of pages! In order to help Google find and index the important ones.
I will run it through Screaming Frog now, just to be sure!
-
hi There,
The following can be reasons for your crawl rate increase
- You have updated the content of the website recently or doing it regularly.
- You / someone from your end submitted the sitemap.xml to google again or doing it over and over.
- Your robots.txt was changed to give access to earlier blocked pages.
- Your or someone used ping services to let search engines know about your website. There are many manual ping services like Pingomatic and in the WordPress you can manually add more ping services to ping many search engine bots. You can find such a list at WordPress ping list post (http://www.shoutmeloud.com/wordpress-ping-list.html).
- You can also monitor and optimize Google Crawl rate using Google Webmaster Tools. Just go to the crawl stats there and analyze. You can manually set your Google crawl rate and increase it to faster or slower. Though I would suggest use it with caution and use it only when you are actually facing issues with bots not crawling your site effectively. You can read more about changing Google crawl rate here https://support.google.com/webmasters/?hl=en&answer=48620#topic=3309469 .
I hope this helps, if you have further queries , feel free to respond.
Regards,
Vijay
-
I wouldn't be concerned at all, have you got one section that expands into a load of other links? It could be that Google hasn't crawled properly for a while and then finds a section they haven't seen before and just goes mad.
Alternatively, have you crawled with screamingfrog or similar tool? Incase there's an issue you weren't aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Links not pointing to my website?
Aloha everyone! I've asked questions to the Moz community before and have always been happy to see how responsive and helpful the community is as a SEO learning new comer! Many thanks in advance for any ideas or insights to solving this problem. So here is my question, I own a wedding photography business here on Kauai and am looking to rank for a few things, Kauai wedding photographer, Kauai Wedding videographer, and Kauai family photographer. That aside I've been pouring more time into getting back links from people I've shot with here on Kauai (local business's websites etc) and have had links dropped in on their sites to, in theory, bump my rankings up from back linking my website. Here's my website http://www.balihaiphoto.com/ Here's a few sites but are not showing up as backlinks/pushing my site higher on SEO results (screen shot as well) http://www.haleleakeiki.org/ (bottom of page footer linking to BaliHaiPhoto.com) and another site here http://www.wishingwellshaveice.com/ with the same thing, footer link pointing to my website. Both sites have been indexed and the shave ice one gets a good amount of traffic, does anyone know why this isn't showing as a back link or if its passing on any link juice my way? Should I do something diffrent here? Let me know what you guys think! Aloha from Kauai, Jon Gibb qsDOM
Reporting & Analytics | | Trey30 -
Backlinks Tracking Websites/Tools/Software
I have multiple websites that I need to keep track of their backlinks. How do you guys keep track of your backlinks? What are some cool tools that you use ?
Reporting & Analytics | | AngelosS0 -
I have conflicting user flow and bounce rates Google Analytics
Has anyone come across this or know how to read this data. In my site content landing page report I have 141 sessions at 0.71% bounce rate = 1 session But then in user flow I have the same page with 123 sessions with a drop off of 117 sessions???
Reporting & Analytics | | Lucas_SOS0 -
How to check bounce rate of mobile or ipad checkout steps? How to check Funnel for mobile or ipad?
Hello Guys, Is there any way to check bounce rate of mobile or ipad checkout steps till thankyou page? Also separate Funnel visualization for mobile or ipad? John
Reporting & Analytics | | varo0 -
High bounce rate on affiliate portal
Hi everyone, I own several affiliate websites and i have a really high bounce percentage(61%) and a high exit rate(78,45%). But i don't know the exact meaning of both words: At the homepage of my website people have the possibility to directly click on several banners, so they then going to another website via the affiliate-link, lots of people click these links. So i was wondering is this the reason why my exit rate en high bounce percentage is that high, or could it have another reason? Thank you,
Reporting & Analytics | | iwebdevnl1 -
When i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
when i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
Reporting & Analytics | | fireproductsuk0 -
Mobile Website Analytics Code and Button Tracking / Event Tracking
How to track the action on the mobile version. Action by pressing the "add comment" this code: onclick = "_gaq.push (['_trackEvent', 'comments', 'pressed'])
Reporting & Analytics | | meteorr
- Not suitable for mobile version Please help.0 -
Organic bounce rate after site re-launch
3 months ago a client of mine re-launched theoir web site (after having a lot of work done on it). Since then, many of the SEO indications are good - more non-paid keywords sending search visits, more organic visits overall, more URLs receiving entrances via search, etc. The issue is that their bounce rate has been increasing pretty much EVERY week since. Has anyone seen a similar issue and what could a potential solution be for this? Thanks everyone!
Reporting & Analytics | | CathalOMaoilfhinn0