MSNbot Issues
-
We found msnbot is doing lots of request at same time to one URL, even considering we have caching, it triggers many requests at same time so caching does not help at the moment:
For sure we can use mutex to make sure URL waits for cache to generate, but we are looking for solution for MSN boot.
123.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET //Type-of-Resource/Fun-Stuff HTTP/1.1" 200 0 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
1.253.27.53 [11/Dec/2012:14:15:10 -0600] "GET /Browse//Fun-Stuff HTTP/1.1" 200 6708 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
We found the following solution: http://www.bing.com/community/site_blogs/b/webmaster/archive/2009/08/10/crawl-delay-and-the-bing-crawler-msnbot.aspx
Bing offers webmasters the ability to slow down the crawl rate to accommodate web server load issues.
User-Agent: *
Crawl-Delay: 10
Need to know if it’s safe to apply that. OR any other advices.
PS: MSNBot gets so bad at times that it could trigger a DOS attack – alone! (http://www.semwisdom.com/blog/msnbot-stupid-plain-evil#axzz2EqmJM3er).
-
Yes, you can add that to your robots.txt file and it should slow down the crawl rate. I haven't tested it myself but have seen many instances of it. Let us know how it works out!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Page Content Issue
Hello, I recently solved www / no www duplicate issue for my website, but now I am in trouble with duplicate content again. This time something that I cannot understand happens: In Crawl Issues Report, I received Duplicate Page Content for http://yourappliancerepairla.com (DA 19) http://yourappliancerepairla.com/index.html (DA 1) Could you please help me figure out what is happenning here? By default, index.html is being loaded, but this is the only index.html I have in the folder. And it looks like the crawler sees two different pages with different DA... What should I do to handle this issue?
Technical SEO | | kirupa0 -
IP Redirect causing Indexing Issue
Hi, I am trying to redirect any IP from outside India that comes to Store site (https://store.nirogam.com/) to Global Store site (https://global.nirogam.com/) using this methodThis is causing various indexing issues for Store site as Googlebot from US also gets redirected!- Very few pages for "store.nirogam.com/products/" are being indexed. Even after submission of sitemap it indexed ~50 pages and then went back to 1 page etc. Only ~20 pages indexed for now.- After this I tried manually indexing via "Crawl -> Fetch as Google" - but then it showed me a redirect to global.nirogam.com. All have their "status -> Redirected" - This is why bots are not able to index the site.What are possible solutions for this? How can we tell bots to index these pages and not get redirected?Will a popup method where we ask user if they are outside India help in solving this issue?All approaches/suggestions will be highly appreciated.
Technical SEO | | pks3330 -
Ranking and Indexing Issue
We have an established site www.getinspired365.com that previously wasn't SEO optimised. We are currently in the process of testing out some new pages to see if we can get them to rank in Google, however we are seeing huge fluctuations in where they rank. Within the first few days we saw our page rank on the first or second page, however it has now dropped out of the top 250 search results. We are wondering if we have made any mistakes with our optimisation ? Example Page : Keyword to target - "If you laugh, you think, and you cry, that's a full day. That's a heck of a day. You do that seven days a week, you're going to have something special." URL : http://www.getinspired365.com/if-you-laugh-you-think-and-you-cry-thats-a-full-day-thats-a-heck-of-a-day-you-do-that-seven-days-a-week-youre-going-to-have-something-special We can see it has been indexed by Google but is now not ranking in the top 250 search engine results. We have run the On Page Grader from SEOMoz and it ranks the page as an "A" so we suspect that we are doing the SEO ok on the page, but can't work out why it isn't ranking, despite ranking on the first or second page after a few days ? We have other pages that aren't SEO optimised that rank better than our newly SEO optimised pages e.g. Keyword - "THE BEST LOVE IS THE KIND THAT AWAKENS THE SOUL AND MAKES US REACH FOR MORE, THAT PLANTS A FIRE IN OUR HEARTS AND BRINGS PEACE TO OUR MINDS. AND THAT'S WHAT YOU'VE GIVEN ME. THAT'S WHAT I'D HOPED TO GIVE YOU FOREVER" URL: http://www.getinspired365.com/20130528 Any advice you could offer would be great. Thanks ! Mike
Technical SEO | | MichaelWhyley0 -
Duplicate content or Duplicate page issue?
Hey Moz Community! I have a strange case in front of me. I have published a press release on my client's website and it ranked right away in Google. A week after the page completely dropped and it completely disappeared. The page is being indexed in Google, but when I search "title of the PR", the only results I get for that search query are the media and news outlets that have reported the news. No presence of my client's page. I also have to mention that I found two URLs of the same page: one with lower case letters and one with capital letters. Is this a duplicate page or a duplicate content issue coming from the news websites? How can I solve it? Thanks!
Technical SEO | | Workaholic0 -
Duplicate content issues, I am running into challenges and am looking for suggestions for solutions. Please help.
So I have a number of pages on my real estate site that display the same listings, even when parsed down by specific features and don't want these to come across as duplicate content pages. Here are a few examples: http://luxuryhomehunt.com/homes-for-sale/lake-mary/hanover-woods.html?feature=waterfront http://luxuryhomehunt.com/homes-for-sale/lake-mary/hanover-woods.html This happens to be a waterfront community so all the homes are located along the waterfront. I can use a canonical tag, but I not every community is like this and I want the parsed down feature pages to get index. Here is another example that is a little different: http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=without-pool http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=4-bedrooms http://luxuryhomehunt.com/homes-for-sale/winter-park/bear-gully-bay.html?feature=waterfront So all the listings in this community happen to have 4 bedrooms, no pool, and are waterfront. Meaning that they display for each of the parsed down categories. I can possible set something that if the listings = same then use canonical of main page url, but in the next case its not so simple. So in this next neighborhood there are 48 total listings as seen at: http://luxuryhomehunt.com/homes-for-sale/windermere/isleworth.html and being that it is a higher end neighborhood, 47 of the 48 listings are considered "traditional listings" and while it is not exactly all of them it is 99%. Any recommendations is appreciated greatly.
Technical SEO | | Jdubin0 -
Http & https canonicalization issues
Howdyho I'm SEOing a daily deals site that mostly runs on https Versions. (only the home page is on http). I'm wondering what to do for canonicalization. IMO it would be easiest to run all pages on https. But the scarce resources I find are not so clear. For instance, this Youmoz blog post claims that https is only for humans, not for bots! That doesn't really apply anymore, right?
Technical SEO | | zeepartner0 -
Identified blog issues, anyone able to help?
hi everyone, I have been going through a site recently and i am noticing certain seo errors that are being caused by the blog. Nothing too harmful, but nonetheless i am hoping to correct them. 1 - The seomoz software have identified that i have duplicate title tags on the following http://www.altman.co.uk/blog http://www.altman.co.uk/blog?page=1 http://www.altman.co.uk/blog?page=2 ETC, ETC, ETC... Now am i right that i need to canonicalize those types of urls with the rel="next" and rel="previous"? <colgroup><col width="583"></colgroup>
Technical SEO | | AITLtd
| <colgroup><col width="583"></colgroup>
| <colgroup><col width="583"></colgroup>
| | | |0 -
How to Fix Duplicate Content Issue of Manufacturer Details Paragraph?
I am surviving with Google's crawling issue. Google had not index my product pages yet. I have Google a lot and read too many articles to get it done. But, I did not get satisfy answer with it. I just checked my product pages and found that: There is one tab with Manufacturers Details containing one paragraph. This content is available on too many product pages with same manufacturer. You can know more by visiting following URL. http://www.vistastores.com/indoorlighting-elklighting-d1472.html So, Does it matter to stop my crawling? If yes so How can I fix it?
Technical SEO | | CommercePundit0