How to authenticate Moz crawler so that others don't use Rogerbot useragent to scrape data from our site?
-
Is there any way to authenticate genuine Moz crawler. Because, our website keeps getting scrapping attacks and if there is no way to authenticate Moz crawler, then, any scraper can just set user agent as Rogerbot and scrape all our pages.
Is there a fixed IP that can be used or any other customization that will help us authenticate and allow only Moz crawler to crawl our site.
Looking forward to a solution to this problem. We haven't been able to use Moz crawler due to this issue.
-
Hi There,
Thanks for writing us so there seems to be a few things going on here so if you need any additional clarification please let me know. So Moz will use a dynamic IP, so there is not just one IP we can provide for authentication.
Unfortunately, your best course of action in this case would be to authorize Mozilla/5.0 (compatible; rogerBot/1.0) This would need to be conducted on the hosting level so you would need to work with your current hosting provider for a viable solutions.
Also, you could set up your robots.txt file to disallow all robots except for Google and Rogerbot, unfortunately malicious robots will often ignore robots.txt files, so any long term solutions would need to go through your hosting provider.
I am sorry that we could not provide more assistance on this matter and hopefully the attacks on your site do not last.
Have a great day!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What does Location National means in Moz Pro Campaigns?
I understand the concept of locations in Moz Pro Campaigns. What does National exactly means? Is it like globally for a United States Google Search Rank or what? Please explain.
Getting Started | | ds9.tech1 -
Why is Moz.com saying that none are linking to www.oneworldcetner.eu
It still says 0 after 6 weeks if i use other tooks like http://openlinkprofiler.org/r/oneworldcenter.eu#.VhbF7ivzJ8E I do see the backlinks that i was counting on was there
Getting Started | | onewordcenter0 -
How do i stop Moz from indexing my dev site?
Hey, I am new to MOZ, and I am seeing in my page reports that I have duplicate content, but this duplicate content is mostly my development site. dev.domain.com. I have this blocked in robots.txt for google. How do i stop MOZ including it in its reports? Cheers everyone.
Getting Started | | Tholomew1
Bart0 -
Moz not reading Visit to my website
Hello , i have conencted moz to My new ecommerce website , MOZ is not reading any visits , although i can see the visit on google analytics and i have moz connected to My analytics as well, why i cant see the visits ? need your help please 🙂
Getting Started | | moeayyad0 -
New to MOZ, can't create a campaign.
I just started the free trial today, but I can't setup a campaign. Everywhere I go (http://analytics.moz.com/pro/home, http://analytics.moz.com/manage-campaigns), all I see is: Oops! Try refreshing the page, if that doesn't work, please click here to contact our help team. Is something broken?
Getting Started | | jcsilkey0 -
I can't export reports in pdf
Hi,I try to export pdf report for my client but this function doesn't work. I tried in different browsers and on different machines - still doesn't work. What am I doing wrong? Thanks in advance,JJ
Getting Started | | jjtech0 -
Can I use wildcards "*" when setting up a new Moz campaign?
Basically I would like the Moz crawler to focus on a specific section of our domain. We do not bucket things via folder groups, so the use of wildcards would be applicable to us. Our URL structure: www.domain.com/some-stuff-here/p12345 Is the example below a valid input to track the above URL structure? www.domain.com//p Thanks.
Getting Started | | WEB-IRS0