Why might my websites crawl rate....explode?
-
Hi Mozzers,
I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically:
9/5/16 - 923
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice. -
Thank you. The site has approx 40 sitemaps (one for each main category). They were submitted by myself a little over a year ago, so I'm not convinced it could be related to this? As a sidenote, when i did submit the sitemaps, there was no difference whatsover in crawl rate/pages indexed, which left me feeling a little disappointed!
I have spent the last year removing pages that didn't need to exist (down from 1.5 mill to 110,000). The site has been cleaned up a lot! But nothing substantial recently.
I should also now add that since 9/18 it has gone back down to a more "normal" crawl rate of approx 1500 per day.
I have no idea what caused it! Thanks for your help and advice though, all appreciated!
-
Hi Silkstream!
One of the only thing I can think of is if someone submitted the sitemap.xml multiple times. This could definitely be a reason that you are see such a drastic increase—literally overnight—in the number of pages being indexed.
I would also check your robots.txt file to see if anything has changed. Is it now allowing access to pages that were previously blocked?
Hope this helps!
-
Did you get an influx of new backlinks? Did you create more pages (are there more indexed pages in GSC)? Did you make any on page optimizations?
There appears to be an update rolling out which usually means increased crawler activity.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
When my new website went live I saw a huge spike in traffic for 3 weeks but then it halved again. Why would this happen?
My new website went live on 13th April and the organic traffic levels increased by 110%, staying at that level for 3 weeks. Then exactly 3 weeks later the organic traffic halved again, going back to the same level as the old site. Has anyone seen this happen before or worked out why?! Thanks Viki P2R76O9
Reporting & Analytics | | TWB0 -
Time For New Website To Rank?
We've been working on a site for the past couple months and going to be launching in a couple weeks. How long does it take for Google to establish a "stabilized" ranking for the site and various pages? The main homepage will have its targeted keywords and each product page will have its own targeted keywords. We have about 100 total pages for the site. I know rankings may fluctuate initially but trying to get an idea of when the rankings are stabilized before we start working on the ongoing SEO like earning backlinks, changing on-page optimization, etc to be able to track changes in rankings over time.
Reporting & Analytics | | vikasnwu0 -
Identifying conversion rate for product
Hi, I need to identify the conversion rate for a product, lets call it a spanner. I have 100s of spanner product urls and I ensured that the url protocols must include the product name e.g /red_spanner so its easy for me to work out the conversion rate in analytics for all my spanner pages as I just add 'spanner' to the landing page filter, hit the ecommerce tab and bingo. What I cant figure out is how to work out the conversion rate for all spanner sales which includes alot of sales which didnt originate on spanner pages e.g. home page > search result > checkout. Theres 1000s of variations of this e.g. email > home > search > product page > checkout. How can I work out the total conversion rate for spanners which needs to include: people landing on a spanner page and people who didnt arrive at a spanner page and did a search with eventually got them to the spanner pages. Hopefully its not as complicated as I think! Thanks in Advance!
Reporting & Analytics | | AndyMacLean0 -
Run Crawl Diagnostics
hi i have fix some error refering the error list how to re-run crawl diagnostics immediate again to check the error ? thanks
Reporting & Analytics | | AlfredLim0 -
Do i need a new dedicated server to increase my website speed
Hi, i have been talking to my hosting company about my site. I am having major problems with the speed of the site. My site is www.in2town.co.uk Ever since i had to redesign my site after a major mistake was made by the hosting company, my site has been running slow and i have tried everything to sort this out including moving to a dedicated server. The trouble is nothing is working and now my hosting companny have told me that i need a new dedicated server which will make it faster. My site is in joomla and the hosting company have told me that the dedicated server below will make the site run faster, but shall i trust them or find another hosting company. Intel i3 540 3.06 Ghz HT 4MB S-Cache $219/mo $289/month500GBStorage6GBRAM10TBBandwidthI am using the following to test the speed of my site http://tools.pingdom.com/fpt/#!/r0spOGObd/www.in2town.co.uk and http://gtmetrix.com/reports/www.in2town.co.uk/kVV1mTDcThe trouble i have is, when you try loading the home page it is slow and when you try moving around the site it is slow.Can anyone please give me some advice.
Reporting & Analytics | | ClaireH-1848860 -
High bounce rate on affiliate portal
Hi everyone, I own several affiliate websites and i have a really high bounce percentage(61%) and a high exit rate(78,45%). But i don't know the exact meaning of both words: At the homepage of my website people have the possibility to directly click on several banners, so they then going to another website via the affiliate-link, lots of people click these links. So i was wondering is this the reason why my exit rate en high bounce percentage is that high, or could it have another reason? Thank you,
Reporting & Analytics | | iwebdevnl1 -
Strange bounce rate trending
what would make it jump up/down so suddenly like that? bouncerate.png
Reporting & Analytics | | adriandg0 -
How to track clicks and "impressions" on a certain botton on my website.
I would like to track the amount of impressions on all pages in a "sub category" doman.com/subcategory/all-impressions-to-these-pages and clicks to a certain button for a contact form. I know that I can add snipets to my analytics code but I'm not sure how to and witch snipet to include. Is it possible?
Reporting & Analytics | | SuperlativB0