Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Top Keywords Ranking Going Down (continuously)
hi,
White Hat / Black Hat SEO | | seotoolshero06
I have an affiliate website where most of my articles are about product Reviews. And it was all going good before 15 february. After that Most of my keywords that were in top 3 places in SERP loses their position.
One of the keyword that was hurted badly after is "best coffee beans".
I have checked the search console and their is no warning issues or something suspicious. I have improved the onpage again, fixing every issue i got from audit tools, from speed to optimize images, titles to alt text, in short everything i could or that can be improved. But still coffee beans keyword is going down every passing day.
I am so worried and want you guys to please help me regarding this. Thanks2 -
Key word Ranking are going down
Hi, my website's Keywords Ranking are going down, I am very new to SEO, I don't know what's the reason please help me out....my website is www.tourspackagedubai.com. Thanks
White Hat / Black Hat SEO | | tariqsalam0 -
Killed by penguin 3
So with the update to penguin 3.0 last week we notice that some clients have been significantly hit by the update. How do we rectify the situation for the poor links that are on the site. We have used open site explorer and Google webmaster to try and identify which are the bad links to try and remove. Now we can spot that some inbound links are from directories that may be perceived as low value/spam, but could not be sure what is affecting the ranking. The vast majority of these links are historical prior to inheriting this client recently and so do not have any logins to remove the links (if there are logins). These appear to be placed by teams outsourced in India. We would suspect that no site owner would spend the time removing links from the site any way. How do we recover from the penguin hit. Is it just a case of trying to identify ones that we suspect could be perceived as spam and ask for these to be disavowed by Google? Do we contact all the sites to ask them to be removed and/or do we just push ahead with more engaging white hat methods of social SEO? Are we likely to recover in the short term or be permanently hit. The site is for a small business with no more than 800 monthly hits so this fall from grace off very good front page positions is going to hit our client very hard even if the sins are from a previous business. Any thoughts and suggestions PLEASE HELP
White Hat / Black Hat SEO | | smartcow0 -
Publishing the same article content on Yahoo? Worth It? Penalties? Urgent
Hey All, I am currently working for a company and they are publishing exactly the same content on their website and yahoo. In addition to this when I put the same article's title it gets outranked by Yahoo. Isn't against Google guidelines? I think Yahoo also gets more than us since they are on the first position. How do you think should the company stop this practice? Please need urgent responses for these questions. Also look at the attachment and look at the snippets. We have a snippet (description) like the first paragraph but yahoo somehow scans the content and creates meta descriptions based on the search queries. How do they do That?
White Hat / Black Hat SEO | | moneywise_test0 -
All pages going through 302 redirect - bad?
So, our web development company did something I don't agree with and I need a second opinion. Most of our pages are statically cached (the CMS creates .html files), which is required because of our traffic volume. To get geotargeting to work, they've set up every page to 302 redirect to a geodetection script, and back to the geotargeted version of the page. Eg: www.example.com/category 302 redirects to www.example.com/geodetect.hp?ip=ip_address. Then that page 302 redirects back to either www.example.com/category, or www.example.com/geo/category for the geo-targeted version. **So all of our pages - thousands - go through a double 302 redirect. It's fairly invisible to the user, and 302 is more appropriate than 301 in this case, but it really worries me. I've done lots of research and can't find anything specifically saying this is bad, but I can't imagine Google being happy with this. ** Thoughts? Is this bad for SEO? Is there a better way (keeping in mind all of our files are statically generated)? Is this perfectly fine?
White Hat / Black Hat SEO | | dholowiski0 -
Month old site and alreasdy ranks 3 for competitive keyword
I know this individual does this with several sites and then offers them for sale to his competitors. Obviously spammy thru and thru, but how can google reward a site thats not even two months old, with 1900 + links with a ranking of #3 for a highly competitive keyword? Please dont post the actual name or url of the website as we dont want to give him any more credit but this blows my mind as he has done this several times with other sites and never gets penalized. http://tinyurl.com/b9jysa5 Any ideas as to how he can accomplish this besides almost 2000 links in less than 2 months? How is that even remotely natural? I know his other sites have been reported to google but they never did anything about it. Thanks for any feedback.
White Hat / Black Hat SEO | | anthonytjm0 -
Page Rank is 0
Hi. Can you please point me in the right direction concerning a site whose default page has a PR of 0? There does not appear to be any errors in the robots.txt file (that I can tell). When I ran a duplicate content check by searching the title tag and first sentance in quotes it did not return more than 2 sites. When I ran a site: it is reporting 287,000 results. Does this mean that they purchased links and have now been penalized? Or where should I go from here? Thank you for any feedback and assistance.
White Hat / Black Hat SEO | | JulB0 -
Interesting Case Study. Website with 70 PA and 73 DA but PR 0?
http://www.jewelry.hyper-info.com/ This website has no backlinks reported on google, but loads of backlinks across the web. It also doesn't even rank top 50 on its topic keyword [Jewelry Tips]. I would be very wary of building a link on a site like this, but thats my opinion. How would you rate this websites link based on this data?
White Hat / Black Hat SEO | | 13375auc30