The Moz Q&A Forum

    • Forum
    • Questions
    • Users
    • Ask the Community

    Welcome to the Q&A Forum

    Browse the forum for helpful insights and fresh discussions about all things SEO.

    1. SEO and Digital Marketing Forum
    2. Categories
    3. Moz Tools
    4. Moz Pro
    5. Rogerbot blocked by cloudflare and not display full user agent string.

    Moz Q&A is closed.

    After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

    Unsolved Rogerbot blocked by cloudflare and not display full user agent string.

    Moz Pro
    roger-botcrawl error
    1 1 2.0k
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as question
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BB_NPG
      BB_NPG last edited by

      Hi,

      We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
      Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help.

      robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it.

      When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ?

      Thanks
      Ben

      Cloudflare activity log, showing differences in user agent strings
      2022-07-01_13-05-59.png

      1 Reply Last reply Reply Quote 0
      • 1 / 1
      • First post
        Last post

      Got a burning SEO question?

      Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


      Start my free trial


      Explore more categories

      • Moz Tools

        Chat with the community about the Moz tools.

        Getting Started
        Moz Pro
        Moz Local
        Moz Bar
        API
        What's New

      • SEO Tactics

        Discuss the SEO process with fellow marketers

        Content Development
        Competitive Research
        Keyword Research
        Link Building
        On-Page Optimization
        Technical SEO
        Reporting & Analytics
        Intermediate & Advanced SEO
        Image & Video Optimization
        International SEO
        Local SEO

      • Community

        Discuss industry events, jobs, and news!

        Moz Blog
        Moz News
        Industry News
        Jobs and Opportunities
        SEO Learn Center
        Whiteboard Friday

      • Digital Marketing

        Chat about tactics outside of SEO

        Affiliate Marketing
        Branding
        Conversion Rate Optimization
        Web Design
        Paid Search Marketing
        Social Media

      • Research & Trends

        Dive into research and trends in the search industry.

        SERP Trends
        Search Behavior
        Algorithm Updates
        White Hat / Black Hat SEO
        Other SEO Tools

      • Support

        Connect on product support and feature requests.

        Product Support
        Feature Requests
        Participate in User Research

      • See all categories

      • Crawler was not able to access the robots.txt
        andrewrench
        andrewrench
        0
        9
        758

      • Crawling only the Home of my website
        Azurius
        Azurius
        0
        7
        1.8k

      • Replicate rogerbot error for server/hosting provider
        Moving-Web-SEO-Auckland
        Moving-Web-SEO-Auckland
        0
        22
        4.4k

      • What is Linking C-Blocks
        shankar333
        shankar333
        5
        4
        4.2k

      Get started with Moz Pro!

      Unlock the power of advanced SEO tools and data-driven insights.

      Start my free trial
      Products
      • Moz Pro
      • Moz Local
      • Moz API
      • Moz Data
      • STAT
      • Product Updates
      Moz Solutions
      • SMB Solutions
      • Agency Solutions
      • Enterprise Solutions
      • Digital Marketers
      Free SEO Tools
      • Domain Authority Checker
      • Link Explorer
      • Keyword Explorer
      • Competitive Research
      • Brand Authority Checker
      • Local Citation Checker
      • MozBar Extension
      • MozCast
      Resources
      • Blog
      • SEO Learning Center
      • Help Hub
      • Beginner's Guide to SEO
      • How-to Guides
      • Moz Academy
      • API Docs
      About Moz
      • About
      • Team
      • Careers
      • Contact
      Why Moz
      • Case Studies
      • Testimonials
      Get Involved
      • Become an Affiliate
      • MozCon
      • Webinars
      • Practical Marketer Series
      • MozPod
      Connect with us

      Contact the Help team

      Join our newsletter
      Moz logo
      © 2021 - 2026 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
      • Accessibility
      • Terms of Use
      • Privacy

      Looks like your connection to Moz was lost, please wait while we try to reconnect.