What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keywords Dropping Out
Hi In March I've seen a lot of keywords go from being ranked, to dropping out completely. We usually get under 1000 drop in and out each month, but 2,500 dropped out this month. Some are not focus keywords I'd be concerned about, but then some are.... I am trying to work out why and could use some help. We haven't changed anything on the pages & here are some examples: http://www.key.co.uk/en/key/dollies-load-movers-door-skates - previously ranked for 'Dollies' position 6 now unranked http://www.key.co.uk/en/key/sack-trucks - ranked position 11 for 'Folding Sack Truck' now unranked http://www.key.co.uk/en/key/portable-workbenches-trestles - ranked position 7 for portable workbenches now unranked. These pages do still rank for other keywords, so aren't out of Google completely. I'm just trying to identify if we have any other issues Thank you 🙂
Intermediate & Advanced SEO | | BeckyKey0 -
Multiple pages for a Profile
Hi All, We have a Biography website in which we have Celebrity News, Videos, Images etc. We have a main page which shows the biography and sides we show 5 blocks of News, Videos, Images etc and have a More option when clicked the page goes to a page where it shows 100's of news of that celeb or Videos etc This is the main page : sitename.com/celebrityname news: sitename.com/celebrityname/news Videos: sitename.com/celebrityname/videos Images: sitename.com/celebrityname/Images Now these 3 pages have no content in them and when we scan via SEMRUSH it shows as "with duplicate title tags". We have 20K such bio's so 20K * 3 such pages is 60K duplicate title tags How can we deal with such pages? Any help please. Thanks
Intermediate & Advanced SEO | | leengsro870 -
Would it work to place H1 (or important page keywords) at the top of your page in HTML and move lower on page with CSS?
I understand that the H1 tag is no longer heavily correlated with stronger ranking signals but it is more important that Keywords or keyphrases are at the top of a page. My question is, if I just put my important keyword (or H1) toward the top of my page in the HTML and move it towards the middle/lower portion with css position elements, will this still be viewed by Googlebot as important keywords toward the top of my page? QCaxMHL
Intermediate & Advanced SEO | | Jonathan.Smith0 -
Website Does not index in any page?
I created a website www.astrologersktantrik.com 4 days ago and fetch it with google but still my website does not index on google as the keywords I use is with low competition but still my website does not appear on any keywords?
Intermediate & Advanced SEO | | ramansaab0 -
Crawl budget
I am a believer in this concept, showing google less pages will increase their importance. here is my question: I manage a website with millions of pages, high organic traffic (lower than before). I do believe that too many pages are crawled. there are pages that I do not need google to crawl and followed. noindex follow does not save on the mentioned crawl budget. deleting those pages is not possible. any advice will be appreciated. If I disallow those pages I am missing on pages that help my important pages.
Intermediate & Advanced SEO | | ciznerguy2 -
Why does our business directions page rank above business profile page
Hi All, We are having an issue at the moment where our business direction page is ranking above the main business profile page. Our website is zodio.com, similar to Yelp but for South East Asia. An example of each page is below: Business Profile Page - http://www.zodio.com/business/detail/126037914/chowking Business Directions - http://www.zodio.com/business/direction/126037914 On many of our long tail searches for particular businesses, the business directions rank above the business details. Does anyone have any idea of why this would happen? I have researched Yelp and they do not have this issue. A few search examples in Google are as follows (one is in Thai): agonos dental clinic เวิลด์ชาร์มมิ่ง kawanku elektrik I have been rattling my brain and search for answers but cannot find anything. The communities help would be much appreciated. Many Thanks, Neil W
Intermediate & Advanced SEO | | zodiothailand0 -
Pricing Page vs. No Pricing Page
There are many SEO sites out there that have an SEO Pricing page, IMO this is BS. A SEO company cannot give every person the same quote for diffirent keywords. However, this is something we are currently debating. I don't want a pricing page, because it's a page full of lies. My coworker thinks it is a good idea, and that users look for a pricing page. Suggestions? If I had to build one (which I am debating against) is it better to just explain why pricing can be tricky? or to BS them like most sites do?
Intermediate & Advanced SEO | | SEODinosaur0 -
Optimisation change caused a drop
We are a web design company and SEO has never been our main thing but we can do it for clients quite well. We were ranking under web design and our location quite well up until 3 months ago. We didn't really have much on our page for web design but targetted website design instead. Our SEO guy who has proved himself for getting some of our clients to the top of Google under very generic search terms recommended we focus on Web Design, so I changed the site accordingly. 2 days later Google has seen our changes and now we've changed our H1 and copy. However, our ranking has dropped 1 place yet again. Obviously I've now panicked and am stressing (even feeling sick) about what to do. He is away until next week so I can't ask him. Could it be that Google has seen we're targetting the word and pushed us down? Shall I wait for the link building he's doing? My own link building lately seems to knock us down a place with every new link I'm doing (and I'm only adding the odd link - not spamming at all). What's odd is that we're still doing extremely well for keywords that aren't even mentioned in our copy at all really. If you could help or offer advice I'd be very grateful.
Intermediate & Advanced SEO | | sanchez19600