Why would I suddenly start seeing a spike in hits from particular bots (specifically rogerbot, google, bing, and yahoo)?
-
We have seen consistent network traffic over the past month, then starting yesterday, huge spikes in hits (hits as in crawls to pages causing an increase in megabytes downloaded) started coming in from Rogerbot, Google, Bing, and Yahoo. A specific example from Rogerbot is as follows:
-
rogerbot/1.1+(http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help,+rogerbot-crawler+pr2-crawler-104@moz.com)
-
Useragent from the bot
-
IP address: 54.226.73.52
-
Domain / hostname: ec2-54-226-73-52.compute-1.amazonaws.com
-
Physical location: United States flag United States, VA, Ashburn
We've have thought about doing a crawl-delay to prevent these bots from hitting us so hard, but that still doesn't help us answer why this even started in the first place.
Any clue on what may be going on here?
-
-
Hi Kasy, did you get to the bottom of this?
-
- No changes yet; however, it's getting worse on our end, particularly from Yahoo, so we're about to update it to add a line for crawl-delay.
- No known changes have been made with any of these.
- No changes have been made to any of our canonical or noindex tags.
- No, everything is the same.
- The only one that we have consistently crawl the site is Moz. I'm familiar with the other tools, but I haven't used them lately to crawl the site.
-
Hi Kasy, sorry to check the obvious first...
- Have there been any updates to your robots.txt file?
- Have you updated sitemaps? In Robots.txt or Google webmaster tools?
- Have you changed any meta information like canonical tags, noindex tags
- Have you changed any internal links from no-follow to follow?
- Have you got any tools regularly set up to crawl your site as Googlebot? Moz, Screaming Frog, DeepCrawl, Xenu etc.
-
Nothing unusual in any of those areas. GA is normal too. The hits/bots did come all at the same time, but since it started, it's been consistent.
-
Anything strange in your link profile or in your social media profile. Did the hits/bots come all at the same time or are spread evenly within 24 hrs?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I use the Google disavow tool?
Hi I'm a bit new to SEO and am looking for some guidance. Although there is no indication in Webmaster tools that my site is being penalised for bad links, I have noticed that I have over 200 spam links for "Pay Day Loans" pointing to my site. (This was due to a hack on my site several years ago). So my question is two fold. Firstly, is it normal to have spammy links pointing to your site and secondly, should I bother to do anything about it? I did some research into the Disavow tool in Webmaster tools wonder I should use it to block all these links. Thanks
Technical SEO | | hotchilidamo0 -
Keyword rankings for new website good in Yahoo and Bing but no movement in Google?
Hello, This past September I launched a new redesigned website for a client. His old website was a static html site that was many years old and the new website was created using WordPress. With the new design we made sure to use all the proper techniques for SEO. (h1 tages, image names, quality links, page titles, etc.) Plus, all the content is new content written for this site. I've actually launched new sites many times and after a few months usually start seeing keyword ranking improvements from the major search engines. With this particular website I'm seeing improvements in Yahoo and Bing but no movement in Google. I've used Google Webmaster Tools and made sure my sitemap is being submitted, etc. It all seems good, but I can't understand why Yahoo and Bing are working but nothing from Google. My page grades are all A's and B's and Moz isn't showing any big issues. Maybe I need to give it more time? This client is a lawyer and has many websites out there so maybe he's being penalized somewhere I don't know? As I mentioned, I've been doing SEO for about 8 years and have never had this much trouble with Google. I was wondering if you can look at the site and see if there are any glaring issues I'm missing. Website: http://www.arizonamedicalmalpractice.info/ The keyword phrases were are looking at are "Phoenix Medical Malpractice Lawyer", "Phoenix Medical Malpractice Attorney:, "Arizona Medical Malpractice Lawyer & Attorney", etc. I appreciate anyone who takes the time and does a quick look over. Thanks very much, Bill
Technical SEO | | Bill_K0 -
Google Disavow Tool
Some background: My rankings have been wildly fluctuating for the past few months for no apparent reason. When I inquired about this, many people said that even though I haven't received any penalty notice, I was probably affected by penguin. (http://moz.com/community/q/ranking-fluctuations) I recently did a link detox by LinkRemovalTools and it gave me a list of all my links, 2% were toxic and 51% were suspiscious. Should I simply disavow the 2%? There are many sites where is no contact info.
Technical SEO | | EcomLkwd0 -
Do Collections in Shopify create Duplicate Pages according to Google/Bing/Yahoo?
I'm using the e-commerce platform Shopify to host an e-store. We've put our products into different collections. Shopify automatically creates different URL paths to a product in multiple collections. I'm worried that the same product listed in different collections is soon as different pages, and therefore duplicate content by Google/Bing/Yahoo. Would love to get your opinion on this concern! Thanks! Matthew
Technical SEO | | HappinessDigital0 -
Should this site start again on a new domain
Hi We have not done SEO on this site they have used another company who looks like they outsourced and the links have been built by a third party all blog networks and this company have said they cannot get the links removed. Google flagged artificial links on this web site in February and in April it lost over 10000 visitors in a month and its just free falled ever since. The categories have been recreated and no redirects created due to the amount of backlinks from the blog sites to the original category pages but the site is not recovering its down to 1500 visitors a month and used to get 14000 a month. So should my customer ditch the domain and move this site to fresh domain? http://www.kids-beds-online.com Any answers would really be appreciated. thanks Tracy
Technical SEO | | dashesndots0 -
Continued Lack of Google Indexing
I run a baseball site (http://www.mopupduty.com) that is in a very good link neighbourhood. ESPN, The Score, USA Today, MSG Network, The Toronto Star, Baseball Prospectucs, etc etc. New content has not been getting indexed on Google ever since the last update. Site has no dup content, 100% original. I can't think of any spammy links, we get organic links day after day. In the past Google has indexed the site in minutes. It currently has expanded site links within Google search. Bing & Yahoo index the site in minutes. Are there any quick fixes I can make to increase my chance to get indexed by Google. Or just keep pumping out content and hope to see a change in the upcoming future?
Technical SEO | | mkoster1 -
Why do I rank so differently in Bing and Yahoo?
Howdy SEOmozers, I've noticed that I rank very differently in Bing and Yahoo compared to Google (and it’s the bad kind of different) for many of the terms that I'm targeting. Can anyone give some advice on how I can put this right or link me to some good resources about the differences in their ranking algorithms? Thanks Paul
Technical SEO | | pauldoffman1