Yahoo Slurp Bot 3.0 Going Crazy
-
On one of our sites, since the Summer, Yahoo Slurp bot has been crawling our pages at about 5 times a minute. We have put a crawl delay on it and it does not respect our robots.txt. Now the issue is it's triggering javascript (which bots shouldn't) triggering our adsense, ad server, analytics information, etc.
We've thought of banning the bot all together but get a good amount of Yahoo traffic. We've though about programmatic-ly not showing the javascript (ad + analytic) tags but are slightly afraid the Yahoo might consider this cloaking.
What are the best practices to deal with this bad bot.
-
I've searched the web but cannot find a specific support location. Any suggestions or links.
-
Bots do folow javascript links these days, maybe yahoo have jsut started to do so, maybe they are not doing so well at it.
I would contact Yahoo and try and get some answers.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Top Keywords Ranking Going Down (continuously)
hi,
White Hat / Black Hat SEO | | seotoolshero06
I have an affiliate website where most of my articles are about product Reviews. And it was all going good before 15 february. After that Most of my keywords that were in top 3 places in SERP loses their position.
One of the keyword that was hurted badly after is "best coffee beans".
I have checked the search console and their is no warning issues or something suspicious. I have improved the onpage again, fixing every issue i got from audit tools, from speed to optimize images, titles to alt text, in short everything i could or that can be improved. But still coffee beans keyword is going down every passing day.
I am so worried and want you guys to please help me regarding this. Thanks2 -
Malicious links on our site indexed by Google but only visible to bots
We've been suffering from some very nasty black hat seo. In Google's index, our pages show external links to various pharmaceutical websites, but our actual live pages don't show them. It seems as though only certain user-agents see the malicious links. Setting up Screaming Frog SEO crawler using the Googlebot user agent also sees the malicious links. Any idea what could have caused this or how this can be stopped? We scanned all files on our webserver and couldn't find any of malicious links. We've changed our FTP and CMS passwords, is there anything else we can do? Thanks in advance!
White Hat / Black Hat SEO | | SEO-Bas0 -
How authentic is a dynamic footer from bots' perspective?
I have a very meta level question. Well, I was working on dynamic footer for the website: http://www.askme.com/, you can check the same in the footer. Now, if you refresh this page and check the content, you'll be able to see a different combination of the links in every section. I'm calling it a dynamic footer here, as the values are absolutely dynamic in this case. **Why are we doing this? **For every section in the footer, we have X number of links, but we can show only 25 links in each section. Here, the value of X can be greater than 25 as well (let's say X=50). So, I'm randomizing the list of entries I have for a section and then picking 25 elements from it i.e random 25 elements from the list of entries every time you're refreshing the page. Benefits from SEO perspective? This will help me exposing all the URLs to bots (in multiple crawls) and will add page freshness element as well. **What's the problem, if it is? **I'm wondering how bots will treat this as, at any time bot might see us showing different content to bots and something else to users. Will bot consider this as cloaking (a black hat technique)? Or, bots won't consider it as a black hat technique as I'm refreshing the data every single time, even if its bot who's hitting me consecutively twice to understand what I'm doing.
White Hat / Black Hat SEO | | _nitman0 -
Do searchs bot understand SEF and non SEF url as the same ones ?
I've jsut realized that since almost for ever I use to code first my website using the non sef for internal linkings. It's very convenient as I'm sure that what ever will be the final url the link will always be good. ex: website.com/component1/id=1 Before releasing the website I use extensions to make the url user friendly according the choosen strategy. ex: website.com/component1/id=1 -> website.com/article1.html But I just wondered if google consider both urls as the same ones or if it consider just as a 301 redirection. What do you think is the best to do ?
White Hat / Black Hat SEO | | AymanH0 -
Correct way to block search bots momentarily... HTTP 503?
Hi, What is the best way to block googlebot etc momentarily? For example, if I am implementing a programming update to our magento ecommerce platform and am unsure of the results and potential layout/ file changes that may impact SEO (Googlebot continuously spiders our site) How can you block the bots for like 30 mins or so? Thanks
White Hat / Black Hat SEO | | bjs20100 -
Blackhat Winners after Penguin 2.0
I know I'm not the only one that's seen this. After Penguin 2.0 some obvious blackhat SEOed sites flew up in the rankings. There's obviously a hole that hasn't been closed. I'm surprised it's been a month and that hole still hasn't been patched. I have no problem with other legit companies out ranking ours for various keywords. In that case I can feel alright knowing it's just something they were able to do that I wasn't but when I see complete blackhat sites ranking that's a whole different story. Estimated traffic before and after Penguin 2.0: http://goo.gl/gurXt What are they doing that's blackhat? Hidden text - compare the cached version vs. the live http://goo.gl/YYGDK 301ing lots of domains, many irrelevant. http://goo.gl/RjOJu Using a trade marked brand (steelers) - not SEO related but I'm sure the NFL wouldn't be happy. Linking between other domains they own. Notice how spammy these sites are. http://pittsburghwebdevelopment.org/2013/06/23/website-development-firm-website-design-pittsburgh/ http://seoinpgh.com/2013/06/23/website-designer-pittsburgh-affordable-web-design-in-pittsburgh-pa/ They were inflating their social presence. Wanted to show you but looks like twitter already took care of them https://twitter.com/seopittsburgh . Also making client sites link to them . http://pittsburghpaplumbing.com/2013/06/19/pittsburgh-plumbersplumbers-in-pittsburgh-paplumber-pittsburgh/ I've talked to other people and they've seen similar things. Thoughts, opinions? Can you find one good reason why this site would rank well for a competitive phrase?
White Hat / Black Hat SEO | | eyeflow0 -
How Would You Go About Building a Private Link Network?
Assuming you need to build a private link network from scratch, how would you go about doing it? I am not looking for some shady tactic, but rather something that would be white hat, yet will help in our SEO efforts. Thanks in advance.
White Hat / Black Hat SEO | | ConversionChamp0 -
Hi, I found that one of my competitors have zero backlings in google, zero in yahoo but about 50.000 in Bing. How is that possible?
Hi, I found that one of my competitors have zero backlings in google, zero in yahoo but about 50.000 in Bing. How is that possible? I assumed that all search engines would finde the backlinks. Besides that he ranks fair well and better than I do with only a single site and with only one article of content while I have a lot of content and sites. I do not undersdtand why he is ranking better in google, while google assumingly does not see any backlinks of the 50.000 bing is finding. Thx, Dan
White Hat / Black Hat SEO | | docschmitti0