Crawl rate drop
-
Hi Guys,
I have a crawl rate drop in webmastertools and can't figure out way. In the last month I removed a lot o duplicate pages that we don't need anymore, there were at least 1.5 million pages. Can this be a motive?
-
Hi there.
Of course it can! The crawl rate is in pages per day, so if you remove pages (especially 1.5 million), there won't be as much to crawl
Also it can happen, if you have the same static pages and the crawl bot has crawled them all. Google doesn't crawl all pages all the time, they have limited resources. So, if you have launched or updated website recently and now not really updating it, you can see the change in crawl rate.
However, you can change crawl rate for 90 days, if you need them to crawl your website constantly (usually good for websites in process of reconstruction). Here: https://support.google.com/webmasters/answer/48620?hl=en
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rankings Continue To Drop
Hi there I'm at wits end trying to stop the slow bleed in our rankings to our store URL's that started mid March 2017 and continues through to today. I'd appreciate some pointers and hope this will throw up a challenge to someone out there. Here is the background: 1. We run an e-commerce store on Shopify with a blog. The recent ranking decline has been almost entirely on the store URL's (catalogue and product pages) while at the same time we have seen steady growth in search volumes in the blog - this makes me think we are seeing a Penguin4 penalty of some type, because the impact is confined to the store URLs. 2. We received a linked based manual penalty back in 2014 and this was successfully removed within 3 months. We have quite a large disavow file as a result. 3. Shortly after launch of Penguin 4.0 in Sept/Oct 2016 we saw a really nice boost in traffic and ascribed this to being under a previous Penguin algo penalty, now removed. 4. Come March 2017 we see a small but steady weekly drop in rankings for our store URL's only, this steady drop continues through to today and over time has become significant. Approximately a 50% decline in visitor numbers to store URL's only as of today, since March. All of this despite: a. Initially I thought this was a Panda issue (because it seemed to coincide with Panda releases in March and May) so the entire website has been rewritten (during June and July) with thin content removed across the store and the blog. Remaining content has been given a serious content boost, being very careful to watch for over-optimisation, and for keyword cannibalisation. I think I've got this right. There are also no crawl issues being highlighted by Moz Pro or SEMRush site audits. b. Recently discovered, only last week in fact. A very low domain website, trust score (0 and 0) had been copying our blog articles steadily on a weekly basis, starting Oct 2016 (yes same time as Penguin4) and only caught last week (my fault for missing this). These articles were copied verbatim with all links and so generated nearly 400 spammy backlinks to our store URLs (about 30% of all the links we have). I've had all these articles removed from the spammy site via DMCA so none of those links exist anymore (as of 8/14/17). I've also disavowed this domain with Google. Could these spam links be the issue, and Google is still needing to crawl this site to see the links are no longer there? I'm not sure because my understanding is that Penguin4 would have devalued these links to start? c. A general review of links and anchor text. I've used Moz Pro and SEMRush backlink audit (linked to Google Search Console) and have removed all toxic links by contacting web masters and using Google disavow. This included removing any links that I think are causing over optimised anchor text. After disavow, according to SEMRush, we have no toxic backlinks left and only 50 out of 1200 links with "Money" anchor text. This exercise was completed two days ago when the last disavow file was uploaded. However I don't believe there was an issue here before as toxic links were < 1% of all links and exact match "money" anchor text in the region of 5%. d. One potential problem with our backlinks is that we have quite a few high domain/high trust links to our scholarship page with anchor text "official website". The net result is that our "Other" anchor text category is just over 50% of total links - these are mainly educational institutions with .edu domains. e. A review of internal linking. We had some what I would refer to as SEO links, linking all product and collection pages across the store, through a tagging type system. This was removed two days ago as it was probably unnecessary for user experience. Other than this I have two concerns remaining with our internal linking structure. The first is that we have quite a big static navigation on the left margin of our store collection pages. This is not faceted navigation, but static. The second is that we've internally linked from almost every blog to our "key" money page in the store, however with varied and non-money anchor text. f. There is nothing in Google Search Console indicating a problem, no manual actions, no significant HTML improvements, and Google has indexed over 90% of URL's compared to the sitemap. All broken links have been fixed - there were a lot before but all fixed as of three weeks ago. g. Checking site speed in GA. Speed has remained constant over the period and we have put in some fixes to improve it. Site speed has not got worse and scores average in Googles speed checker. That's about it. It's possible that with the recent changes made with respect to b, c, e and f above I just need to wait a couple more weeks for Google to catch up, and would appreciate thoughts on this. However I'd also like some thoughts on the static navigation on our collection pages, plus importantly on linking from blog articles to mostly a single money page in the store - of all that remains I think this is potentially a problem. Our website is located at www.thekewlshop.com Many thanks for your help. Charles
Intermediate & Advanced SEO | | charlesfitz0 -
Drop in Ranking for a Reputable Dictionary / April 14, 2017
One of the sites of my customers (a very reputable, high quality dictionary) saw a 46% drop in rankings starting from April 14, 2017. Has anyone else seen something similar and has some tips? Many thanks, Tatiana
Intermediate & Advanced SEO | | lascribacchina0 -
Website keeps dropping in ranking with no visible penalty.
Hi everyone! I would really appreciate your help on this! The website URL is: www.gipsydharma.com It has now been over a year, but we're finding it very difficult to start ranking for any useful keywords. Some keywords like: leather boots / boots for women / handmade clothing at some point went up to page 20, but have now dropped again below page 50. Now, most of our traffic comes from social media so the business is going OK. But the question still remains, what does Google have against the above URL? Its not a super amazing website, but it provides unique and engaging content and has zero spam. I also don't think its over optimized, but I may be wrong on this. Recently, I've also noticed that the Domain Authority has also been going down, it was 46 at some point and now its 40. There's no visible penalty and all the pre-Penguin links (of which there weren't that many, I think) were cleaned up months ago, without Google suggesting that. There are very few keyword rich backlinks and a lot of them come naturally anyways. So again, any advice would be greatly appreciated. Thanks in advance for your help!
Intermediate & Advanced SEO | | GipsyDharma0 -
Can changing G+ authorship on a well-ranking article drop its search ranking?
We have an article that ranks #1 in Google SERP for the keyword we want it to rank for. We decided to revise the article because although it's performing well, we knew it could be better and more informative for the user. Now that we've revised the content, we're wondering: Should we update the article author (and the G+ authorship markup) to reflect that the revisor authored the content, or keep the original author listed? Can changing G+ authorship on an article impact its search ranking, or is that an issue that's a few Google algorithm updates down the road?
Intermediate & Advanced SEO | | pasware0 -
SEOmoz is only crawling 2 pages out of my website
I have checked on Google Webmaster and they are crawling around 118 pages our of my website, store.itpreneurs.com but SEOmoz is only crawling 2 pages. Can someone help me? Thanks Diogo
Intermediate & Advanced SEO | | jslusser0 -
Webmaster Index Page significant drop
Has anyone noticed a significant drop in indexed pages within their Google Webmaster Tools sitemap area? We went from 1300 to 83 from Friday June 23 to today June 25, 2012 and no errors are showing or warnings. Please let me know if anyone else is experiencing this and suggestions to fix this?
Intermediate & Advanced SEO | | datadirect0 -
Why does Google Claimed Local Listing Ranking Drop?
I have two local google places listinggs unlaimed. Both listings were ranking in the blended search in 7 pack. Once I claimed the local listings for the business both listings rankings have dropped. And one has totally vanished from the search rankings. Is this normal as it appears local places that are not claimed are ranking higher than local places claimed?
Intermediate & Advanced SEO | | VivaArturo0 -
Sudden drop in page rank
Hi One of our client websites has had a sudden drop in rankings, and also in page rank. We have obviously been off for 2 weeks, so havent been blogging daily etc as we normally would have. Would this cause such a drop? Some keywords have lost from position 2/3 to page 7 over night, and we havent change the strategy! Thanks,
Intermediate & Advanced SEO | | SEOwins0