Crawl rate drop
-
Hi Guys,
I have a crawl rate drop in webmastertools and can't figure out way. In the last month I removed a lot o duplicate pages that we don't need anymore, there were at least 1.5 million pages. Can this be a motive?
-
Hi there.
Of course it can! The crawl rate is in pages per day, so if you remove pages (especially 1.5 million), there won't be as much to crawl
Also it can happen, if you have the same static pages and the crawl bot has crawled them all. Google doesn't crawl all pages all the time, they have limited resources. So, if you have launched or updated website recently and now not really updating it, you can see the change in crawl rate.
However, you can change crawl rate for 90 days, if you need them to crawl your website constantly (usually good for websites in process of reconstruction). Here: https://support.google.com/webmasters/answer/48620?hl=en
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl and Indexation Error - Googlebot can't/doesn't access specific folders on microsites
Hi, My first time posting here, I am just looking for some feedback on a indexation issue we have with a client and any feedback on possible next steps or items I may have overlooked. To give some background, our client operates a website for the core band and a also a number of microsites based on specific business units, so you have corewebsite.com along with bu1.corewebsite.com, bu2.corewebsite.com. The content structure isn't ideal, as each microsite follows a structure of bu1.corewebsite.com/bu1/home.aspx, bu2.corewebsite.com/bu2/home.aspx and so on. In addition to this each microsite has duplicate folders from the other microsites so bu1.corewebsite.com has indexable folders bu1.corewebsite.com/bu1/home.aspx but also bu1.corewebsite.com/bu2/home.aspx the same with bu2.corewebsite.com has bu2.corewebsite.com/bu2/home.aspx but also bu2.corewebsite.com/bu1/home.aspx. Therre are 5 different business units so you have this duplicate content scenario for all microsites. This situation is being addressed in the medium term development roadmap and will be rectified in the next iteration of the site but that is still a ways out. The issue
Intermediate & Advanced SEO | | ImpericMedia
About 6 weeks ago we noticed a drop off in search rankings for two of our microsites (bu1.corewebsite.com and bu2.corewebsite.com) over a period of 2-3 weeks pretty much all our terms dropped out of the rankings and search visibility dropped to essentially 0. I can see that pages from the websites are still indexed but oddly it is the duplicate content pages so (bu1.corewebsite.com/bu3/home.aspx or (bu1.corewebsite.com/bu4/home.aspx is still indexed, similiarly on the bu2.corewebsite microsite bu2.corewebsite.com/bu3/home.aspx and bu4.corewebsite.com/bu3/home.aspx are indexed but no pages from the BU1 or BU2 content directories seem to be indexed under their own microsites. Logging into webmaster tools I can see there is a "Google couldn't crawl your site because we were unable to access your site's robots.txt file." This was a bit odd as there was no robots.txt in the root directory but I got some weird results when I checked the BU1/BU2 microsites in technicalseo.com robots text tool. Also due to the fact that there is a redirect from bu1.corewebsite.com/ to bu1.corewebsite.com/bu4.aspx I thought maybe there could be something there so consequently we removed the redirect and added a basic robots to the root directory for both microsites. After this we saw a small pickup in site visibility, a few terms pop into our Moz campaign rankings but drop out again pretty quickly. Also the error message in GSC persisted. Steps taken so far after that In Google Search Console, I confirmed there are no manual actions against the microsites. Confirmed there is no instances of noindex on any of the pages for BU1/BU2 A number of the main links from the root domain to microsite BU1/BU2 have a rel="noopener noreferrer" attribute but we looked into this and found it has no impact on indexation Looking into this issue we saw some people had similar issues when using Cloudflare but our client doesn't use this service Using a response redirect header tool checker, we noticed a timeout when trying to mimic googlebot accessing the site Following on from point 5 we got a hold of a week of server logs from the client and I can see Googlebot successfully pinging the site and not getting 500 response codes from the server...but couldn't see any instance of it trying to index microsite BU1/BU2 content So it seems to me that the issue could be something server side but I'm at a bit of a loss of next steps to take. Any advice at all is much appreciated!0 -
Positions dropped and do not recover
Hi all, Site: https://www.fascinators.com.au k/w: fascinators (this is the main one I am looking at) About 2 years ago, I redesigned the site. It used to be on first positions in Google Australia. After redesign, positions dropped and never recovered. No matter what I do, I can not get them back. I even involved professional SEO agency for the last 6 month to help build links, but nothing works. (old site used really old non mobile-friendly engine) As far as I know site is technically sound with no issues. In the last few month position for my main keyword 'fascinators' jumps really wildly. From position 9 to not being in Google first 100. And jump can happen overnight. As of today it disappeared from first 100 again. I am really tearing my hair out on what can be wrong. Link building is underway. This is one area I am aware of. Thinking of keyword spamming, i removed lots of 'fascinators' from URL, but that did not seem to help much. Any ideas will be greatly appreciated! Thanks, Rudolf
Intermediate & Advanced SEO | | rudolfl1 -
DA dropped 7 pts in one week as well as our competitors. Is something going on?
We have consistently had a 70 in DA and now all of the sudden we are at a 63 as well as seeing the same considerable drop among our competitors. This was odd as I have not seen this type of drop in DA since being with the company. Any ideas are appreciated.
Intermediate & Advanced SEO | | WeaverMike0 -
Crawl efficiency - Page indexed after one minute!
Hey Guys,A site that has 5+ million pages indexed and 300 new pages a day.I hear a lot that sites at this level its all about efficient crawlabitliy.The pages of this site gets indexed one minute after the page is online.1) Does this mean that the site is already crawling efficient and there is not much else to do about it?2) By increasing crawlability efficiency, should I expect gogole to crawl my site less (less bandwith google takes from my site for the same amount of crawl)or to crawl my site more often?Thanks
Intermediate & Advanced SEO | | Mr.bfz0 -
Drop in rankings
Our keyword rankings have dropped greatly since July'12. Our site used to be in the top 8 rankings for the most competitive keywords on our industry and now rank 30+. Being new to this industry any help would be greatly appreciated. www.fragileremovals.com.au
Intermediate & Advanced SEO | | RobSchofield0 -
Why are Pages returning 404 errors not being dropped?
Our webmaster tools continues to return anywhere upwards of 750 pages that have 404 errors. These are from pages of a previous site no longer used. However this was over 1 year ago these pages were dropped along with the 301 re-directs. Why is Google not clearing these from webmaster tools but re-listing them again after 3 month cycle? Is it because external sites have links to these pages? If so should I put a 301 in place (most of these site are forums and potentially dodgy directories etc from previous poor link building programs) or ask for a manual removal?
Intermediate & Advanced SEO | | Towelsrus0 -
Would this drop indicate a manual penalty?
Website short link: f c w . i m (copy and remove the spaces) A few weeks ago now we dropped from around page 2 all the way to around page 14 for they keyword watches on Google UK. We have remained around the level of page 12-17 ever since. Other important keywords which we monitor have slowly moved from page 1 positions onto page 2 or the bottom of page 1. Of course this is really worrying us as we are an e-commerce website and we are in peak season. Natural suspects would be duplicate content issues, crawl issues or bad links. All of which we have looked into and spent the past month improving to the best of our ability. I have gone through almost all of the content on the website. We have our own written descriptions on our 5000 products and have identified a small amount with issues using Copyscape. We have lots of unique customer product reviews and we have our own unique blog. I have looked into Crawl Issues and fine tuned URL parameter settings, usage of canonical and added next and prev tags. All of the faceted navigation which shouldn't be indexed has been excluded through canonical for well over a month and again recently using URL parameters in WT. Our link profile is small and doesn't contain a lot of spam links - we have identified some and wish to get them removed but even so I don't think the small quantity of links (a lot of which are nofollow also) would justify dropping us over around 100 places for a clearly relevant keyword. The only other thing that might be an issue is a large number of on page links. This is partly due to drop down page navigation. All our pages are being indexed by Google though so I'm not sure if it is a problem. You could argue it dilutes page rank, but you would think Google's algorithms would take recurring page navigation into account somehow - removing it would probably be detrimental to our users. So really we wanted to see if any SEO experts could help me out with this. It seems to us that it is either something we have already identified (causing a lot more impact than we would expect following the latest Google updates) or something else. Maybe a manual penalty? Thanks if you read the whole thing! Didn't intend to write this much really!
Intermediate & Advanced SEO | | Scott.lucas1 -
Does Bing(and Yahoo)Crawl AJAX Based Content?
I found this article and Bing appeared to at this time have a checkbox for the option for AJAX handling although looking at the Bing Webmaster Tools it doesnt appear that this option is available. Has it just been completely integrated now, relieving webmaster from needing to check the option or is it no longer supported? http://searchengineland.com/bing-now-supports-googles-crawlable-ajax-standard-84149
Intermediate & Advanced SEO | | imiJoe0