A Serious drop in Pages crawled per day
-
On 21st April ,I spotted a sudden decrease in pages crawled per day.Previously it was about 5,000 bust after the drop it reached to 225.From the crawl rate never spiked.
Here is my website url - http://www.wpstuffs.com/
-
What strikes me is at the same time as your pages crawled fell off a cliff your page load time doubles. What changed round about then that would have caused an increase in load times?
If that was around the time you switched CDN, I'd be double and triple checking the new one was set up perfectly and is an improvement on cloudflare. Did all your pages suddenly get markedly bigger, perhaps due to another plugin being added? Pingdom tools is also great for picking apart load times.
That would be where I'd start...
-
Iam using WordPress blogging platform and checked the GWT and Analytics but there were no break in code.
Im using Google Pagespeed CDN service ? will that cause such problems.Previously I was using Cloudflare.
Current PageSpeed is 92 and load time is under 3seconds.I checked the pingdom report and it doesn't show any down time stats.
-
Yes, well there are a couple of things to consider and test.
First and foremost, check your GWT code and make sure that nothing's change. Same with Google Analytics.
Have you changed platforms?
If not, I would say it's "Hoy Crap Batman" time and you need to start analyzing everything. What's your Pagespeed? Is your site online? If so, was it offline earlier in the week? (Maybe you weren't checking every minute) If your site is hosted by someone else, get on the phone and demand some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
AMP pages - should we be creating AMP versions of all site pages?
Hi all, Just wondering what people's opinions are on AMP pages - having seen the Google demo of how AMP pages will be given visibility on page one of Google for news-based content, do you think it is worth considering creating AMP versions of all pages, ready for when Google expands its inclusion of these super-fast pages?
Algorithm Updates | | A_Q1 -
Reasons for a sharp decline in pages crawled
Hello! I have a site I've been tracking using Moz since July. The site is mainly stagnant with some on page content updates. Starting the first week of December, Moz crawler diagnostics showed that the number of pages crawled decreased from 300 to 100 in a week. So did the number of errors through. So crawler issues went from 275 to 50 and total pages crawled went from 190 to 125 in a week and this number has stayed the same for the last 5 weeks. Are the drops a red flag? Or is it ok since errors decreased also? Has anyone else experienced this and found an issue? FYI: sitemap exists and is submitted via webmaster tools. GWT shows no crawler errors nor blocked URLs.
Algorithm Updates | | Symmetri0 -
Ranking drops POST Penguin?!
Hey guys, I've noticed that one of our sites have seen some considerable ranking drops for competitive keywords at least a week or so after Penguin was introduced. I've attached an image to give you a better idea of what I'm talking about. We were ranked around 2nd page for each of the phrases but plummeted to 80+ (although, there has been some movement back towards 50). I mention Penguin but that really shouldn't have been a factor with this site as it's a relatively new one and all the links have been built naturally. Any help appreciated! GGsL8xk.png
Algorithm Updates | | Webrevolve0 -
Google local page
I was in google places and then set up a goole local account 9 months ago but it is not showing as listed. What am I doing wrong or what can I do about it?? https://plus.google.com/u/0/106415232585067786390/about thanks Jo Ann
Algorithm Updates | | jojojo0 -
So, useless link exchange pages still work?!
After 3 years out of SEO I thought things might have moved on, but apparently not. Bit of back link research and all the top sites in my niche have tons of reciprocal links to barely relevant sites. Do I really have to do this? I mean I thought this was so out of date, it's not much better than keyword stuffing. So, should I just forget my lofty principles asking myself 'is this of any value to my users?' and just take the medicine?
Algorithm Updates | | Cornwall0 -
Google Dropped 3,000+ Pages due to 301 Moved !! Freaking Out !!
We may be the only people stupid enough to accidentally prevent the google bot from indexing our site. In our htaccess file someone recently wrote the following statement RewriteEngine On
Algorithm Updates | | David_C
RewriteCond %{HTTP_HOST} ^mysite.com$ [NC]
RewriteRule ^(.*)$ http://www.mysite.com/$1 [L,R=301] Its almost funny because it was a rewrite that rewrites back to itself... We found in webmaster tools that the site was not able to be indexed by the google bot due to not detecting the robots.txt file. We didn't have one before as we didn't really have much that needed to be excluded. However we have added one now for kicks really. The robots.txt file though was never the problem with regard to the bot accessing the site. Rather it was the rewrite statement above that was blocking it. We tested the site not knowing what the deal was so we went under webmaster tools then health and then selected "Fetch as Google" to have the website. This was our way of manually requesting the site be re-indexed so we could see what was happening. After doing so we clicked on status and it provided the following: HTTP/1.1 301 Moved Permanently
Content-Length: 250
Content-Type: text/html
Location: http://www.mystie.com/
Server: Microsoft-IIS/7.5
MicrosoftOfficeWebServer: 5.0_Pub
MS-Author-Via: MS-FP/4.0
X-Powered-By: ASP.NET
Date: Wed, 22 Aug 2012 02:27:49 GMT
Connection: close <title>301 Moved Permanently</title> Moved Permanently The document has moved here. We changed the screwed up rewrite mistake in the htaccess file that found its way in there but now our issue is that all of our pages have been severely penalized with regard to where they are now ranking compared to just before the indecent. We are essentially freaking out because we don't know the real time consequences of this and if or how long it will take for the certain pages to regain their prior ranks. Typical pages when down anywhere between 9-40 positions on high volume search terms. So to say the least our company is already discussing the possibilities of fairly large layoffs based on what we anticipate with regard to the drop in traffic. This sucks because this is peoples lives but then again a business must make money and if you sell less you have to cut the overhead and the easiest one is payroll. I'm on a team with three other people that I work with to keep the SEO side up to snuff as much as we can and we sell high ticket items so the potential effects if Google doesn't restore matters could be significant. My question is what would you guys do? Is there any way we can contact Google about such a matter? If you can I've never seen such a thing. I'm sure the pages that are missing from the index now might make their way back in but what will there rank look like next time and with that type of rewrite has it permanently effected every page site wide, including those that are still in the index but severely effected by the index. Would love to see things bounce back quick but I don't know what to expect and neither do my counterparts. Thanks for any speculation, suggestions or insights of any kind!!!0 -
How could Penguin kill my top ten rank and promote this garbage page to a #5 spot
Hey, Before penguin, I had a #9 rank for the term "yoga poses". So as many of us are doing, I started looking at my link profile... and yes, there were around 300 links from an old yoga news website (anchor: yoga poses)... that lead to the page on my site optimized for this term. The problem is they took the site down, but not properly... I.E. they generate a "not available" message for browsers, but underneath, I guess the bots can still index all the pages... so I guess they were interpreting these links as coming from a cloaked site. So, I was able to get them to remove the links... webmaster tools reports half of them gone now. What I don't get though... is how Google can give this garbage page a #5 spot for a competitive term like "yoga poses"... Check out http://www.ebmyoga.com/beginyoga.html and compare it to my page... http://www.yogaclassplan.com/yoga-poses/ This page leads to highly quality 100% unique yoga pose articles... in my mind we deliver so much more value than the site with a #5 rank. I don't understand. Any insight? Thanks,
Algorithm Updates | | biomat0 -
Perfect Landing Page for jewelry website?
What would be the Perfect Landing Page for jewelry website? Show me some examples and tell me why do you think so. Thanks in advance to everyone who participated.
Algorithm Updates | | DiamondJewelryEmpire0