Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Negative SEO Click Bot Lowering My CTR?
I am questioning whether one of our competitors is using a click bot to do negative SEO on our CTR for our industry's main term. Is there any way to detect this activity? Background: We've previously been hit by DoS attacks from this competitor, so I'm sure their ethics/morals wouldn't prevent them from doing negative SEO. We sell an insurance product that is only offered through broker networks (insurance agents) not directly by the insurance carriers themselves. However, our suspect competitor (another agency) and insurance carriers are the only ones who rank on the 1st page for our biggest term. I don't think the carrier sites would do very well since they don't even sell the product directly (they have pages w/ info only) Our site and one other agency site pops onto the bottom of page one periodically, only to be bumped back to page 2. I fear they are using a click bot that continuously bounces us out of page 1...then we do well relatively to the other pages on page 2 and naturally earn our way back to page 1, only to be pushed back to page 2 by the negative click seo...is my theory. Is there anything I can do to research whether my theory is right or if I'm just being paranoid?
White Hat / Black Hat SEO | | TheDude0 -
Why have bots (including googlebot) categorized my website as adult?
How do bots decide whether a website is adult? For example, I have a gifting portal, but strangely here, it is categorized as 'Adult'. Also, my google adsense application to run ads on my site got rejected - I have a feeling this is because googlebot categorized my site as adult. And there are good chances that other bots also consider it an adult website, rather than a gifting website. Can anyone please go through the site and tell me why this is happening? Thanks in advance.
White Hat / Black Hat SEO | | rahulkan0 -
Link profile heavy with press release syndication links caused drop at Penguin 2.0
I'm wrestling with something that I'm hoping members of the community can provide input on.... I've working with an enterprise level client that is in the business of data capture and distribution. I've diagnosed a clear drop of traffic on May 22nd, i.e a loss of search visibility post Penguin 2.0. Their link profile is big! Discussions with internal stakeholders who have been with the company 10's of years confirm that no "link building" service providers have ever been hired and no over-zealous employee is ever likely to have tried to "do" link building internally. They are just one of those lucky companies that by their nature publish information that people want to link to and share. As a first port of call I've grouped links by anchor text and can see groups of hundreds of matching anchors based on their brand URL and specific page titles. The matching anchors have resulted from big take up of interesting data that they have marketed via press releases. NOT for link purposes. My question is this.... Does the community think or have evidence (or can point me toward any case studies) that show that Press release syndication alone could result in: a) a penguin penalty or...
White Hat / Black Hat SEO | | QubaSEO
b) a devaluing of press release type links during Penguin 2.0 that could have resulted in a loss of search visibility and give the impression of a penalty Your thoughts are much appreciated!0 -
Are Links from blogs with person using keyword anchor text a Penguin 2.0 issue?
Hello, I am continuing a complete clean up of a clients link profile and would like to know if Penguin is against links from blogs with the user including keywords as anchor text? So far I have been attempting to get them removed before I go for a disavow. An example would be the work clothing comment at the bottom of: http://www.fashionstyleyou.co.uk/beat-the-caffeine-rush.html/comment-page-1 I am also questioning if we should keep any link directories, so far I have been ruthless, but worry I will be losing a hell of a lot of links. For example I have kept the following: http://www.business-directory-uk.co.uk//clothing.htm Your comments are welcomed!
White Hat / Black Hat SEO | | MarzVentures0 -
Penalized by Penguin 2.0
I believe our site has been penalizes by Penguin 2.0. Our impressions in Google Webmaster are down and our traffic in Google Analytics also took a hit. Both of these occurences took place right when Penguin 2.0 was unleashed. What are the steps I need to take to regain my ranking? Is disallowing all the links I think maybe spammy the first thing to do?
White Hat / Black Hat SEO | | joebuilder0 -
Old SPAM tactic still works and gets TOP 3 in SERP?
Hi Mozers, Below you can see some examples of spam ( hidden text and sneaky redirects) which are in SERP for our branded keywords during last 3 months. Some of them occupy very high position in SERP (top 3/top5). https://www.google.com/search?num=100&newwindow=1&safe=off&biw=1883&bih=1028&q=%22your+mac+-%22%2B%22cleanmymac%22 I sent spam reports and I’m going to continue doing so. (~500 spam reports from personal and work google account) I contacting directly with some of the hacked sites (web-masters) and tried to help them to fix this issue, but it takes a lot of my time. But 3 months!? Can you give me any advice, what doing next? Thank you!
White Hat / Black Hat SEO | | MacPaw0 -
One of my outbound links website go hit by panda!
Hi mozzers, today I received a message from one of my blogger partners announcing me that he got hit by panda. 2 weeks ago I had him placing 2 anchors one in our main domain and a second one on our subdomain. I know panda focuses essentially on dups and I have paid attention to our webmaster tools to make sure we haven t got any messages Which we re good with. What do you guys suggest, will this affect us at some point or we re good? also in case that we re good will panda affect the blogger's authority therefore ours? if yes I should probably remove them, right? Thanks
White Hat / Black Hat SEO | | Ideas-Money-Art0 -
What do Bing and Yahoo look for in a site?
Do Bing and Yahoo look for authoritative sites like google does? Do they punish sites for black hat or spamming? The reason I ask these questions is because one of my competitors was ranking in first place for many great keywords in Google, they have the highest authority out of all of their competitors. They must have been punished by Google because now they are not ranking for any great keywords in Google. However they are ranking 1st in Bing and Yahoo for most the top keywords, getting the the most visibility out of all the sites. I attached a small Graph with latest visibility for the sites with the top keywords from google and then I also included the company that was punished from google they are the green circles on the graph. dIVI4.png
White Hat / Black Hat SEO | | SEODinosaur0