undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Pro
  4. Is there a whitelist of the RogerBot IP Addresses?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Is there a whitelist of the RogerBot IP Addresses?

Moz Pro
5
8
5.9k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • EricCholis
    EricCholis last edited by Mar 17, 2011, 11:32 AM

    I'm all for letting Roger crawl my site, but it's not uncommon for malicious spiders to spoof the User-Agent string. Having a whitelist of Roger's IP addresses would be immensely useful!

    1 Reply Last reply Reply Quote 1
    • chill986
      chill986 @AaronWheeler last edited by Oct 11, 2016, 2:23 PM Oct 11, 2016, 2:23 PM

      Samantha (of the Moz team) suggested I have my client whitelist Rogerbot - so you are saying simply whitelist Rogerbot as a useragent?  Is there any other information I need to provide?

      1 Reply Last reply Reply Quote 0
      • topic:timeago_earlier,3 years
      • cleanteam
        cleanteam last edited by Mar 31, 2014, 11:13 AM Mar 31, 2014, 11:13 AM

        Gotcha thanks for the response, Aaron.

        1 Reply Last reply Reply Quote 1
        • AaronWheeler
          AaronWheeler @cleanteam last edited by Mar 29, 2014, 5:53 PM Mar 29, 2014, 5:53 PM

          Hey Kalen! Rogerbot is the crawler we use to gather data on websites for Moz Analytics and the Mozscape link index. Here's his info: http://moz.com/help/pro/what-is-rogerbot-.

          I wish I could give you IP addresses, but they change all the time since we host Roger in the cloud. There's not even a reliable range of IPs to give you. You can totally whitelist the useragent rogerbot, but that's the only reliable information about the crawler you can go off of. I hope that helps but let me know if there's any other solution you can think of. Thank you!

          chill986 1 Reply Last reply Oct 11, 2016, 2:23 PM Reply Quote 1
          • cleanteam
            cleanteam last edited by Mar 27, 2014, 6:01 PM Mar 27, 2014, 6:01 PM

            Hi Aaron,

            I'm not totally sure what RogerBot is, but I was also interested in a list of IPs to white list.  We just completed a search crawl and are checking out the Crawl Diagnostics.  It's hit some 503 errors b/c it's triggering our DoS filter.

            Is there a way to get the IP addresses behind this crawl in order to white list them?

            Thanks,
            Kalen

            AaronWheeler 1 Reply Last reply Mar 29, 2014, 5:53 PM Reply Quote 0
            • topic:timeago_earlier,3 years
            • AaronWheeler
              AaronWheeler last edited by Mar 18, 2011, 6:50 PM Mar 18, 2011, 6:50 PM

              Hey there Outside!

              I totally understand your concerns, but unfortunately we don't have a static IP we can give you for Rogerbot. He's crawling from the cloud so his IP address changes all the time! As you know, you can allow him in Robots.txt but that's the only way to do it for now. We have a recent post about why this may be risky business: http://www.seomoz.org/blog/restricting-robot-access-for-improved-seo

              Hope that helps!

              1 Reply Last reply Reply Quote -1
              • EricCholis
                EricCholis @scanlin last edited by Mar 17, 2011, 2:56 PM Mar 17, 2011, 2:56 PM

                Personally, I've run across spiders that search for entry points and exploits in common CMS, e-commerce, and CRM web applications. For example, there was a recent Wordpress bug that could be exploited to serve malicious content (read: virus) to visiting users.

                Spoofing the User-Agent string is elementary at best, and wouldn't fool any sys admin worth a salt. All you have to do is a WHOIS on the requested IP to help identify it's origin.

                I'm a bit of a data geek, so I like to grep through log files to see things that won't show up in Analytics that require Javascript.

                1 Reply Last reply Reply Quote 0
                • scanlin
                  scanlin last edited by Mar 17, 2011, 2:43 PM Mar 17, 2011, 2:43 PM

                  Out of curiosity (and because I don't know), what is the advantage for a malicious spider to spoof the User-Agent string? I mean, I understand this hides its identity, but why does a spider need to hide its identity? And what can a malicious spider do that a browsing human can't do? I haven't taken any action to prevent robots from anything on my site. Should I?

                  EricCholis 1 Reply Last reply Mar 17, 2011, 2:56 PM Reply Quote 0
                  • 1 / 1
                  1 out of 8
                  • First post
                    1/8
                    Last post

                  Got a burning SEO question?

                  Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


                  Start my free trial


                  Browse Questions

                  Explore more categories

                  • Moz Tools

                    Chat with the community about the Moz tools.

                  • SEO Tactics

                    Discuss the SEO process with fellow marketers

                  • Community

                    Discuss industry events, jobs, and news!

                  • Digital Marketing

                    Chat about tactics outside of SEO

                  • Research & Trends

                    Dive into research and trends in the search industry.

                  • Support

                    Connect on product support and feature requests.

                  • See all categories

                  Related Questions

                  • Skites2

                    Unsolved DA dropping after spam attack

                    moz pro da domain authority spam backlink attack spam backlinks

                    Hello. Our DA has dropped from 40 to 28. Why does moz take into account clear spam links. Not all links are created by website owners. Popular websites also see a lot of unknown spam linking. Any solutions to over-ride these issues? It seems very futile to see the DA dropping without any apparent reason or help towards actionable data.

                    Moz Pro | Nov 30, 2024, 4:10 PM | Skites2
                    0
                  • paulavervo

                    Unsolved The Moz.com bot is overloading my server

                    rogerbot moz crawler

                    The Moz.com bot is overloading my server. How to solve it?

                    Moz Pro | Oct 5, 2024, 8:43 AM | paulavervo
                    0
                  • BB_NPG

                    Unsolved Rogerbot blocked by cloudflare and not display full user agent string.

                    roger-bot crawl error

                    Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
                    Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
                    Ben Cloudflare activity log, showing differences in user agent strings
                    2022-07-01_13-05-59.png

                    Moz Pro | Jul 1, 2022, 12:14 PM | BB_NPG
                    0
                  • ikuraa

                    Unsolved ip address

                    ipaddress nodomain

                    hi, im currently running my web without domain, can i use moz with my web ip address. thanks

                    Moz Pro | Dec 10, 2021, 8:35 PM | ikuraa
                    0
                  • randomghrytu

                    How to increase DA and PA of my site.

                    I have been working on my site for the last 4 months and still, I am not getting improvement in site DA, PA, and site ranking.  Also, I am generating unique content.  Here is the Website link https://techynewx.com/

                    Moz Pro | Jun 11, 2024, 4:49 PM | randomghrytu
                    0
                  • gary_nectarbridge

                    How crucial are H1 tags and descriptions in wordpress categories?

                    Hi all Trying to improve SEO for my (mostly) local site, www.nectarbridge.com, and recently got back on Moz Pro account. First crawl of my site by Moz, a manageable number of issues that I've mostly sorted, but the category with the largest number of problems is missing or invalid tags. My content pages and blog posts are not missing the tags. It's category, archives, etc., including multiple pages, ex: https://www.nectarbridge.com/category/blog/page/4/ A smaller number of pages are being flagged by Moz as missing descriptions, and they are also category pages and the like. So the question is - how hard should I pursue fixing these issues? I'm using the divi theme, which apparently doesn't display the category description by default (if it did, that would kill two birds with one stone). There is a fix to add the category description, but before I get into that I'm trying to discern whether this issue really matters greatly to SEO or if I should spend that time just working on more content.

                    Moz Pro | Jun 24, 2024, 11:19 AM | gary_nectarbridge
                    0
                  • sfchronicle

                    SEO Web Crawler IP addresses

                    What are the IP addresses for the SEO Web Crawler? There is a firewall on my clients website before it goes live, I would like to crawl the site before it goes live, but need to provide the web crawlers IP addreses. Thank you for your time

                    Moz Pro | Oct 10, 2012, 6:37 PM | sfchronicle
                    1
                  • richardstrange

                    Is having the company's address in the footer (or header) of each webpage important for SEO?

                    Is having the company/office's address in the footer (or header) of each webpage important for SEO? Your page for the Geotarget tool says that having the address in this element helps search engines find your location. My question is, how important or relevant is this to SEO? How does knowing the address influence SEO? Is it best SEO practice to put the address in the footer of every webpage? http://www.seomoz.org/geotarget

                    Moz Pro | Jul 2, 2011, 4:32 AM | richardstrange
                    0

                  Get started with Moz Pro!

                  Unlock the power of advanced SEO tools and data-driven insights.

                  Start my free trial
                  Products
                  • Moz Pro
                  • Moz Local
                  • Moz API
                  • Moz Data
                  • STAT
                  • Product Updates
                  Moz Solutions
                  • SMB Solutions
                  • Agency Solutions
                  • Enterprise Solutions
                  Free SEO Tools
                  • Domain Authority Checker
                  • Link Explorer
                  • Keyword Explorer
                  • Competitive Research
                  • Brand Authority Checker
                  • Local Citation Checker
                  • MozBar Extension
                  • MozCast
                  Resources
                  • Blog
                  • SEO Learning Center
                  • Help Hub
                  • Beginner's Guide to SEO
                  • How-to Guides
                  • Moz Academy
                  • API Docs
                  About Moz
                  • About
                  • Team
                  • Careers
                  • Contact
                  Why Moz
                  • Case Studies
                  • Testimonials
                  Get Involved
                  • Become an Affiliate
                  • MozCon
                  • Webinars
                  • Practical Marketer Series
                  • MozPod
                  Connect with us

                  Contact the Help team

                  Join our newsletter
                  Moz logo
                  © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
                  • Accessibility
                  • Terms of Use
                  • Privacy

                  Looks like your connection to Moz was lost, please wait while we try to reconnect.