undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Getting Started
  4. What is the full user-agent for rogerbot?

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

What is the full user-agent for rogerbot?

Getting Started
3
5
7.4k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • uShip
    uShip Subscriber last edited by Jan 29, 2015, 1:05 PM

    IT is blocking AWS via a proxy in front of our server.  We've tried allowing the "roberbot" user-agent but crawling functionality still isn't working in my Moz Pro account.  Is there a more specific user-agent we can allow in our proxy software?  Thank you.

    1 Reply Last reply Reply Quote 1
    • SarahWJATL
      SarahWJATL @uShip last edited by Nov 24, 2015, 7:55 PM Nov 24, 2015, 7:55 PM

      We are having this same issue. I was hoping someone clarified it for you.

      Did you ever get it sorted out?

      1 Reply Last reply Reply Quote 0
      • BlueprintMarketing
        BlueprintMarketing @uShip last edited by Feb 4, 2015, 5:52 AM Feb 4, 2015, 5:52 AM

        User-agent: *

        User-agent: dotbot
        Disallow: /
        User-agent: rogerbot
        Disallow: /

        if you want to prevent robots from crawling your site truly prevent them you will need to use either a password restriction or a tool similar to this

        http://www.distilnetworks.com/

        If you see what is being said by Google and Moz a robots.txt file can not guarantee   blocking something that is linked to. if you want to do that you will have to block the referral using a WAF like distilnetworks

        http://moz.com/help/guides/search-overview/crawl-diagnostics

        https://moz.com/researchtools/ose/dotbot

        &

        https://support.google.com/webmasters/answer/6062608?rd=2

        Also blocking link analysis user agents that are nothing but a drain on your resources is a good idea. Simple enough to do in htaccess with something like this:

        Search Engine Blocked by Robots.txt

        This page cannot be crawled by search engines due to the robots.txt protocol. If you're seeking to remove this page from search results, we recommend that you use meta robots (with noindex, follow values) instead of robots.txt. This will ensure that the page does not appear in the results but allows link juice to flow through the page's links and count towards the relevance/popularity of other pages on your site.

        How to block DotBot from crawling your site

        If you don't want dotbot crawling your site, we always respect the standard Robots Exclusion Protocol (aka robots.txt). If you would like to block dotbot, all you need to do is add our user-agent string to your robots.txt file.

        If you want to ban dotbot from most areas of your site, it looks a little something like this:

              User-agent: dotbot
              Disallow: /admin/
              Disallow: /scripts/
              Disallow: /images/
            
        

        below this I have placed what somebody has created that they state works I do not know if it works I told you that distill networks will work but I cannot guarantee the very bottom I think you will not have any trouble if you set up the robots.txt as configured at the top.

        If you want to ban dotbot from crawling any part of your site, add this text instead:

              User-agent: dotbot
              Disallow: /
        
            
        

        BEGIN

        RewriteEngine On
        RewriteCond %{HTTP_USER_AGENT} ^rogerbot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^exabot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^MJ12bot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^dotbot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^gigabot [OR]
        RewriteCond %{HTTP_USER_AGENT} ^AhrefsBot
        RewriteRule .* – [F]

        SetEnvIfNoCase User-Agent .rogerbot. bad_bot
        SetEnvIfNoCase User-Agent .exabot. bad_bot
        SetEnvIfNoCase User-Agent .mj12bot. bad_bot
        SetEnvIfNoCase User-Agent .dotbot. bad_bot
        SetEnvIfNoCase User-Agent .gigabot. bad_bot
        SetEnvIfNoCase User-Agent .ahrefsbot. bad_bot
        SetEnvIfNoCase User-Agent .sitebot. bad_bot

        Order Allow,Deny
        Allow from all
        Deny

        END

        Thomas

        1 Reply Last reply Reply Quote 0
        • uShip
          uShip Subscriber @BlueprintMarketing last edited by Jan 30, 2015, 2:32 PM Jan 30, 2015, 2:32 PM

          Thank you for your response but this doesn't answer my question.  We aren't blocking rogerbot using robots.txt.  We need to allow it through the proxy in front of our web server by using the exact user-agent (case sensitive) that is being sent by rogerbot.  We've tried "rogerbot" but that isn't working. Based on the 3rd party documentation you linked to there seem to be a variety of possibilities:

          roberbot/1.0
          roberBot/1.0
          RogerBot/1.0
          rogerBot
          RogerBot

          It would be great if Moz provided clear documentation on this.

          BlueprintMarketing SarahWJATL 2 Replies Last reply Nov 24, 2015, 7:55 PM Reply Quote 1
          • BlueprintMarketing
            BlueprintMarketing last edited by Jan 29, 2015, 8:08 PM Jan 29, 2015, 1:27 PM

            HiI know there are two crawlers that Moz uses Roger bot and open site Explorer uses dotbot

            Make sure there is no forward slash "/" after e.g. (Disallow:/)

            Mozused to have an awesome writeup on it but it just forwards to Moz.com/help now it could be that they have another great write up but the URL changed. For now here's the information

            User-agent: rogerbot
            Disallow: 
            User-agent: dotbot
            Disallow:

            User Agent Analyser
            Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

            http://www.useragentstring.com/Dotbot1.1_id_16014.php

            https://udger.com/resources/ua-list/bot-detail?bot=rogerbot

            http://www.botopedia.org/user-agent-list/crawlers/item/369-rogerbot-seomoz

            hope this helps,

            Thomas

            uShip 1 Reply Last reply Jan 30, 2015, 2:32 PM Reply Quote 1
            • 1 / 1
            1 out of 5
            • First post
              1/5
              Last post

            Got a burning SEO question?

            Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


            Start my free trial


            Browse Questions

            Explore more categories

            • Moz Tools

              Chat with the community about the Moz tools.

            • SEO Tactics

              Discuss the SEO process with fellow marketers

            • Community

              Discuss industry events, jobs, and news!

            • Digital Marketing

              Chat about tactics outside of SEO

            • Research & Trends

              Dive into research and trends in the search industry.

            • Support

              Connect on product support and feature requests.

            • See all categories

            Related Questions

            • HFM97

              Unsolved How can I improve SEO for my auditing firm’s website?

              website website optimization

              Hi Moz community! I run aauditing.com, offering audit, tax consulting, and accounting services in the UAE. I’m looking for effective SEO strategies to boost visibility and attract clients. Any tips on content optimization, backlinks, or local SEO would be appreciated!

              Getting Started | Nov 11, 2024, 10:53 AM | HFM97
              2
            • laurentjb

              Unsolved How can I shorten a url?

              url issue url

              I've got way too many long url's but I have no idea how to shorten them?

              Getting Started | Oct 25, 2024, 8:02 AM | laurentjb
              0
            • andrewrench

              Unsolved Crawler was not able to access the robots.txt

              robots.txt crawl error

              I'm trying to setup a campaign for jessicamoraninteriors.com and I keep getting messages that Moz can't crawl the site because it can't access the robots.txt. Not sure why, other crawlers don't seem to have a problem and I can access the robots.txt file from my browser. For some additional info, it's a SquareSpace site and my DNS is handled through Cloudflare. Here's the contents of my robots.txt file: # Squarespace Robots Txt User-agent: GPTBot User-agent: ChatGPT-User User-agent: CCBot User-agent: anthropic-ai User-agent: Google-Extended User-agent: FacebookBot User-agent: Claude-Web User-agent: cohere-ai User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile User-agent: AdsBot-Google-Mobile-Apps User-agent: * Disallow: /config Disallow: /search Disallow: /account$ Disallow: /account/ Disallow: /commerce/digital-download/ Disallow: /api/ Allow: /api/ui-extensions/ Disallow: /static/ Disallow:/*?author=* Disallow:/*&author=* Disallow:/*?tag=* Disallow:/*&tag=* Disallow:/*?month=* Disallow:/*&month=* Disallow:/*?view=* Disallow:/*&view=* Disallow:/*?format=json Disallow:/*&format=json Disallow:/*?format=page-context Disallow:/*&format=page-context Disallow:/*?format=main-content Disallow:/*&format=main-content Disallow:/*?format=json-pretty Disallow:/*&format=json-pretty Disallow:/*?format=ical Disallow:/*&format=ical Disallow:/*?reversePaginate=* Disallow:/*&reversePaginate=* Any ideas?

              Getting Started | Sep 2, 2024, 4:25 PM | andrewrench
              0
            • depawl52

              Unsolved Website Traffic

              website traffic organic search

              Greetings All. I'm working on a new business website for a client, and I've been accessing the site numerous times daily (troubleshooting, confirming changes, etc.). I've been using Google search to access the site, and I use a VPN so that my IP would be random. So I would presume that the site traffic should be increasing. But on the last Moz Pro crawl, the site traffic was still listed as 0.
              Is there a minimum amount of traffic required before Moz recognizes it or is something else going on?
              Thank you.

              Getting Started | May 30, 2024, 10:49 AM | depawl52
              0
            • KapitiGirl

              Unsolved Domain Overview - no NZ?

              domainoverview

              Why on Domain Overview are there only 4 countries to choose from? I am in NZ and need to check websites that operate businesses in NZ not Canada, US or Australia?

              Getting Started | Apr 5, 2024, 8:06 PM | KapitiGirl
              0
            • verdet3232

              Using the free domain analysis tool - what would cause "Bummer no data found"

              When I enter my domain in the free analysis tool, I get a "bummer, no data found". I am certain whatever is causing that to happen is causing other SEO problems https://academicanv.org

              Getting Started | Oct 7, 2024, 4:03 PM | verdet3232
              3
            • MyFamilClubLtd

              Moz not able to crawl our site - any advice?

              When I try and crawl our site through Moz it gives this message: Moz was unable to crawl your site on Aug 7, 2019. Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster. I have been through all the help and doesn't seem to be any issues. You can check the site and robots.txt here: https://myfamilyclub.co.uk/robots.txt. Anyone got any advice on where I could go to get this sorted?

              Getting Started | Jun 27, 2024, 8:47 PM | MyFamilClubLtd
              1
            • Stevepair

              Has anyone purchased the MOZ SEO courses? Are they good?

              moz academy

              I'm looking to learn more about SEO. Has anyone purchased the courses available through MOZ? If so, are they useful?

              Getting Started | Apr 8, 2024, 4:41 PM | Stevepair
              0

            Get started with Moz Pro!

            Unlock the power of advanced SEO tools and data-driven insights.

            Start my free trial
            Products
            • Moz Pro
            • Moz Local
            • Moz API
            • Moz Data
            • STAT
            • Product Updates
            Moz Solutions
            • SMB Solutions
            • Agency Solutions
            • Enterprise Solutions
            Free SEO Tools
            • Domain Authority Checker
            • Link Explorer
            • Keyword Explorer
            • Competitive Research
            • Brand Authority Checker
            • Local Citation Checker
            • MozBar Extension
            • MozCast
            Resources
            • Blog
            • SEO Learning Center
            • Help Hub
            • Beginner's Guide to SEO
            • How-to Guides
            • Moz Academy
            • API Docs
            About Moz
            • About
            • Team
            • Careers
            • Contact
            Why Moz
            • Case Studies
            • Testimonials
            Get Involved
            • Become an Affiliate
            • MozCon
            • Webinars
            • Practical Marketer Series
            • MozPod
            Connect with us

            Contact the Help team

            Join our newsletter
            Moz logo
            © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
            • Accessibility
            • Terms of Use
            • Privacy

            Looks like your connection to Moz was lost, please wait while we try to reconnect.