Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Increase in Direct Traffic plus Bounce Rate rise for all traffic sources
Hello, I work for an agency and we have seen a big rise in bounce rate for 4 of our clients which happened on the exact same day. This rise on bounce rate is across all traffic sources. We are also seeing a big increase in direct traffic, starting on the same day. Is it possible for bot traffic to affect the bounce rate of all other traffic sources? We have ruled out double reporting in GA but can explain how the bounce rate has increased for all traffic sources. How is this linked to the rise in direct traffic (in some cases as high as 500%)? Thanks
Reporting & Analytics | | jenallen0 -
Will adjusted bounce rate affect avg time on page?
I recently read Rob Beirne's piece on how and why to set up an adjusted bounce rate in Google Analytics (https://moz.com/blog/adjusted-bounce-rate). I am getting myself ready to talk to our site team about why we should set up an adjusted bounce rate and am anticipating some questions I'd like to be able to answer: 1. Will an adjusted bounce rate improve the accuracy of our avg time on page metrics? 2. Are we able to keep the unadjusted bounce rate in GA as well, so we can compare the two metrics if we ever need to? Does anyone know the answers to these questions? Any help would be much appreciated!
Reporting & Analytics | | seoisfun0 -
Backlinks Tracking Websites/Tools/Software
I have multiple websites that I need to keep track of their backlinks. How do you guys keep track of your backlinks? What are some cool tools that you use ?
Reporting & Analytics | | AngelosS0 -
I have conflicting user flow and bounce rates Google Analytics
Has anyone come across this or know how to read this data. In my site content landing page report I have 141 sessions at 0.71% bounce rate = 1 session But then in user flow I have the same page with 123 sessions with a drop off of 117 sessions???
Reporting & Analytics | | Lucas_SOS0 -
Seeing massive spikes in direct traffic with 100% bounce rates
Hi all, Looking through Analytics yesterday, I saw that my website had a huge increase in direct traffic in sessions. However, they apparently spent 0 seconds on the website in total so that raised plenty of red flags. Does anyone have reasons why this might be? Spam or bug? Thanks in advance!
Reporting & Analytics | | Whittie0 -
Identifying conversion rate for product
Hi, I need to identify the conversion rate for a product, lets call it a spanner. I have 100s of spanner product urls and I ensured that the url protocols must include the product name e.g /red_spanner so its easy for me to work out the conversion rate in analytics for all my spanner pages as I just add 'spanner' to the landing page filter, hit the ecommerce tab and bingo. What I cant figure out is how to work out the conversion rate for all spanner sales which includes alot of sales which didnt originate on spanner pages e.g. home page > search result > checkout. Theres 1000s of variations of this e.g. email > home > search > product page > checkout. How can I work out the total conversion rate for spanners which needs to include: people landing on a spanner page and people who didnt arrive at a spanner page and did a search with eventually got them to the spanner pages. Hopefully its not as complicated as I think! Thanks in Advance!
Reporting & Analytics | | AndyMacLean0 -
When i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
when i first add my url to seomoz then i had a general report of all the faults my website had in SEO and suggestions where can i find it now , i cant find it ?!
Reporting & Analytics | | fireproductsuk0 -
How to track clicks and "impressions" on a certain botton on my website.
I would like to track the amount of impressions on all pages in a "sub category" doman.com/subcategory/all-impressions-to-these-pages and clicks to a certain button for a contact form. I know that I can add snipets to my analytics code but I'm not sure how to and witch snipet to include. Is it possible?
Reporting & Analytics | | SuperlativB0