What would cause a drastic drop in pages crawled per day?
-
The site didn't go down.
There were no drop in rankings, or traffic.
But we went from averaging 150,000 pages crawled per day, to ~1000 pages crawled per day.
We're now back up to ~100,000 crawled per day, but we went more than a week with only 1000 pages being crawled daily.
The question is, what could cause this drastic (but temporary) reduction in pages crawled?
-
I wish that were the case, but the site wasn't down.
I looked into the errors, they were redirecting to a subdomain that no longer exists.
-
So several times in one month the entire site couldn't be reached. That's pretty significant. Personally I don't have any clients with that many down-times so can only assume that's the cause or at least a partial cause. And more important, a red flag that would prompt me to find a better hosting provider if it were my site.
-
The drop happened March 28th.
There was a "domain name not found" on march 30th (two more on the 22nd, 18th, 12th, and 10th)
-
There could be several factors. When did it occur? Did you see any other crawl errors reported? And unfortunately, the other unknown comes from the fact that Google's own system is both far from perfect and sometimes crawl volume is affected by their own system.
Unless I see crawl errors or an increase in pages not found during or leading up to that period, or more important, see a corresponding significant drop in organic traffic, personally I just chalk it up to the complexity of the web.
-
Hi Alan!
There were no spikes in kb per day or time spent downloading a page.
-
Fatwallet
Have you checked Google Webmaster Tools for crawl errors and other metrics? I had a client recently who had a severe slowdown in their server network which showed up on page crawl speed time as a huge spike - pages loading five times slower than normal. They subsequently had a dip in pages crawled due to the bottleneck.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can noindexed pages accrue page authority?
My company's site has a large set of pages (tens of thousands) that have very thin or no content. They typically target a single low-competition keyword (and typically rank very well), but the pages have a very high bounce rate and are definitely hurting our domain's overall rankings via Panda (quality ranking). I'm planning on recommending we noindexed these pages temporarily, and reindex each page as resources are able to fill in content. My question is whether an individual page will be able to accrue any page authority for that target term while noindexed. We DO want to rank for all those terms, just not until we have the content to back it up. However, we're in a pretty competitive space up against domains that have been around a lot longer and have higher domain authorities. Like I said, these pages rank well right now, even with thin content. The worry is if we noindex them while we slowly build out content, will our competitors get the edge on those terms (with their subpar but continually available content)? Do you think Google will give us any credit for having had the page all along, just not always indexed?
Intermediate & Advanced SEO | | THandorf0 -
Can a "site split" cause a drastic organic search decline?
Let's say you have a client. They have two big, main product offerings. Come early April of this year, one of the product offerings decide to move their product offering over to a new domain. Let's also say you had maybe 12 million links in your inbound link portfolio for the original domain. And when this product offering that split opened their new domain, they 301 redirected half of those 12 million links (maybe even 3/4s) over to their new domain. So you're left with "half" a website. And while you still have millions of links; you lost millions as well. Would a ~25-50% drop in organic traffic be a reasonable effect? My money is on YES. Because all links to a domain help "rise" the page authority sea level of all URLs of the domain. So cutting off 50-75% of those links would drop that sea level a somewhat corresponding amount. We did get some 301 redirects that we felt were "ours" in place in late July... but that really accounted for 25% of the total amount of pages with inbound links they took originally. And those got in place almost 4 months after the fact. Curious what other people may think. LnEazzi.png
Intermediate & Advanced SEO | | ChristianMKG0 -
Pagination on a product page with reviews spread out on multiple pages
Our current product pages markup only have the canonical URL on the first page (each page loads more user reviews). Since we don't want to increase load times, we don't currently have a canonical view all product page. Do we need to mark up each subsequent page with its own canonical URL? My understanding was that canonical and rel next prev tags are independent of each other. So that if we mark up the middle pages with a paginated URL, e.g: Product page #1http://www.example.co.uk/Product.aspx?p=2692"/>http://www.example.co.uk/Product.aspx?p=2692&pageid=2" />**Product page #2 **http://www.example.co.uk/Product.aspx?p=2692&pageid=2"/>http://www.example.co.uk/Product.aspx?p=2692" />http://www.example.co.uk/Product.aspx?p=2692&pageid=3" />Would mean that each canonical page would suggest to google another piece of unique content, which this obviously isn't. Is the PREV NEXT able to "override" the canonical and explain to Googlebot that its part of a series? Wouldn't the canonical then be redundant?Thanks
Intermediate & Advanced SEO | | Don340 -
Will SEO cause a drop in the number of impressions?
Hello, I have been a member of the Moz community for a long time. I very seldom ask questions here but this time I really need your help to make sure I will not make mistakes that will negatively affect my site. My site monetizes according to the number of impressions visualized by the users who visit it. I now want to try to optimize it by using all those nice SEO techniques I have learned through Moz. My goal is to make sure that if I use the various SEO strategies, I will still be able to obtain the high number of impressions I get now. If not, I prefer to leave the situation untouched and I will not start optimizing the site for SEO. Please kindly read the situation below and give me a little support to make sure I am doing the right thing. I would like to ask for your professional advice to solve and issue related to duplicate content. Please visit my site: www.chhedonna.it. The sitemap has indexed 21.890 articles, but if I digit the command 'site:www.chedonna.it', I obtain 158.000 results.I believe that the duplicated content has emerged due to three errors I would like to indicate in order for you to tell me whether my interpretation is correct or not: The article writers made a mistake in that although the content of the articles is different, they have employed the same title for all of them. Therefore, you can see 5 articles with similar/duplicated Tag Titles but the content of said articles is different. For example, http://www.chedonna.it/attualita/cronaca-rosagossip/2012/12/03/heidi-klum-senza-trucco-e-parrucco-foto and http://www.chedonna.it/attualita/cronaca-rosagossip/2012/12/03/heidi-klum-senza-trucco-e-parrucco-foto-2/ are different articles (i.e., the content of the articles is different from each other) that have been published using the same titles. If I inserted the 'follow-noindex' tag to 3 of the 4 duplicates, as shown in the example above, would that be a solution? I fear that if I did insert the follow-noindex, I would cause a drop in the number of impressions visualized by my site's users. It is important to point out, in fact, that I monetize the site via the number of impressions I generate. Therefore, it is fundamental that I do not compromise the number of impressions that the site gets, if I try to optimize it for SEO reasons. On the other hand, I believe that the idea to operate via a 'rel canonical' would not be right, considering that the content of every post having the same title is different, that is, the articles are different even if they focus on the same topic. Also, I would not find it beneficial to use a '301 redirect', since the number of duplicated Titles Tags is very consistent. 2. The second error concerns the duplicate content due to the images that have been included in the articles. For instance, http://www.chedonna.it/che-donna-di-mondo/fare-la-valigia/2012/08/06/campeggio-vacanza-in-liberta/attachment/tenda/ http://www.chedonna.it/che-donna-di-mondo/fare-la-valigia/2012/08/06/campeggio-vacanza-in-liberta/attachment/tenda-2/ http://www.chedonna.it/che-miss/2012/12/04/tatuaggi-fiore-di-loto-significato-e-foto/attachment/tatuaggio-fiore-di-loto-2/ http://www.chedonna.it/che-miss/2013/03/15/tatuaggi-fiore-di-loto-significato-e-foto-2/attachment/tatuaggio-fiore-di-loto-2-2/. I could solve this problem by preventing the media indexing. But I fear that this would produce a very high number of '404 error' messages. If such a thing did happen, my site would stop monetizing overall and I cannot allow this to occur, as you can understand. My very important question is the following: if I prevent the indexing of the photos, will I get in return a drop in the number of impressions that my site would normally generate? 3. Duplicate content generated by the indexing of archive subpages. For example: http://www.chedonna.it/tag/angelica-e-ferdinando/ http://www.chedonna.it/tag/angelica-e-ferdinando/page2/ http://wwwchedonna.it/tag/angelica-e-ferdinando/page3/ If I prevented the media indexing, will I get in return a drop in the number of impressions and many 404 errors? Thank you very much for taking the time to help me sort out this very important issue. Cheers, Sal
Intermediate & Advanced SEO | | salvyy0 -
Hey guys i have this issues on my crawling report what should i do to exlude the pages? are d
Overly-Dynamic URL Overly-Dynamic URL Although search engines can crawl dynamic URLs, search engine representatives have warned against using over 2 parameters in a given URL. Search engines may also see dynamic versions of the same URL as unique URLs, creating duplicate content.
Intermediate & Advanced SEO | | adulter0 -
Keyword Frequent On and Drop Off
Hi, I was wondering if anyone could help me. I am currently doing SEO for a client’s site, over the past few weeks the keywords have gradually gone up steadily. One of the keywords however continues to jump on and off rankings. For a few days it will jump up to 23<sup>rd</sup> in Google Rankings, but then it will completely drop of the radar, not even in the top 200 anymore. I’ve gone through the site to check if there is anything wrong and stuck to guidelines offered here on SEO Moz but I’m stumped to figure it out. The Keyword has a page that helps target it, there is no stuffing, all I can think is if there are bad back-links coming in to us that Google that continues to push down the rankings. Can anyone help shed some light on the situation for me?
Intermediate & Advanced SEO | | Kal-SEO0 -
Dynamic pages - ecommerce product pages
Hi guys, Before I dive into my question, let me give you some background.. I manage an ecommerce site and we're got thousands of product pages. The pages contain dynamic blocks and information in these blocks are fed by another system. So in a nutshell, our product team enters the data in a software and boom, the information is generated in these page blocks. But that's not all, these pages then redirect to a duplicate version with a custom URL. This is cached and this is what the end user sees. This was done to speed up load, rather than the system generate a dynamic page on the fly, the cache page is loaded and the user sees it super fast. Another benefit happened as well, after going live with the cached pages, they started getting indexed and ranking in Google. The problem is that, the redirect to the duplicate cached page isn't a permanent one, it's a meta refresh, a 302 that happens in a second. So yeah, I've got 302s kicking about. The development team can set up 301 but then there won't be any caching, pages will just load dynamically. Google records pages that are cached but does it cache a dynamic page though? Without a cached page, I'm wondering if I would drop in traffic. The view source might just show a list of dynamic blocks, no content! How would you tackle this? I've already setup canonical tags on the cached pages but removing cache.. Thanks
Intermediate & Advanced SEO | | Bio-RadAbs0 -
What causes internal pages to have a page rank of 0 if the home page is PR 5?
The home page PageRank is 5 but every single internal page is PR 0. Things I know I need to address each page has 300 links (Menu problem). Each article has 2-3 duplicates caused from the CMS working on this now. Has anyone else had this problem before? What things should I look out for to fix this issue. All internal linking is follow there is no page rank sculpting happening on the pages.
Intermediate & Advanced SEO | | SEOBrent0