undefined
Skip to content
Moz logo Menu open Menu close
  • Products
    • Moz Pro
    • Moz Pro Home
    • Moz Local
    • Moz Local Home
    • STAT
    • Moz API
    • Moz API Home
    • Compare SEO Products
    • Moz Data
  • Free SEO Tools
    • Domain Analysis
    • Keyword Explorer
    • Link Explorer
    • Competitive Research
    • MozBar
    • More Free SEO Tools
  • Learn SEO
    • Beginner's Guide to SEO
    • SEO Learning Center
    • Moz Academy
    • SEO Q&A
    • Webinars, Whitepapers, & Guides
  • Blog
  • Why Moz
    • Agency Solutions
    • Enterprise Solutions
    • Small Business Solutions
    • Case Studies
    • The Moz Story
    • New Releases
  • Log in
  • Log out
  • Products
    • Moz Pro

      Your all-in-one suite of SEO essentials.

    • Moz Local

      Raise your local SEO visibility with complete local SEO management.

    • STAT

      SERP tracking and analytics for enterprise SEO experts.

    • Moz API

      Power your SEO with our index of over 44 trillion links.

    • Compare SEO Products

      See which Moz SEO solution best meets your business needs.

    • Moz Data

      Power your SEO strategy & AI models with custom data solutions.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Free SEO Tools
    • Domain Analysis

      Get top competitive SEO metrics like DA, top pages and more.

    • Keyword Explorer

      Find traffic-driving keywords with our 1.25 billion+ keyword index.

    • Link Explorer

      Explore over 40 trillion links for powerful backlink data.

    • Competitive Research

      Uncover valuable insights on your organic search competitors.

    • MozBar

      See top SEO metrics for free as you browse the web.

    • More Free SEO Tools

      Explore all the free SEO tools Moz has to offer.

    NEW Keyword Suggestions by Topic
    Moz Pro

    NEW Keyword Suggestions by Topic

    Learn more
  • Learn SEO
    • Beginner's Guide to SEO

      The #1 most popular introduction to SEO, trusted by millions.

    • SEO Learning Center

      Broaden your knowledge with SEO resources for all skill levels.

    • On-Demand Webinars

      Learn modern SEO best practices from industry experts.

    • How-To Guides

      Step-by-step guides to search success from the authority on SEO.

    • Moz Academy

      Upskill and get certified with on-demand courses & certifications.

    • MozCon

      Save on Early Bird tickets and join us in London or New York City

    Unlock flexible pricing & new endpoints
    Moz API

    Unlock flexible pricing & new endpoints

    Find your plan
  • Blog
  • Why Moz
    • Small Business Solutions

      Uncover insights to make smarter marketing decisions in less time.

    • Agency Solutions

      Earn & keep valuable clients with unparalleled data & insights.

    • Enterprise Solutions

      Gain a competitive edge in the ever-changing world of search.

    • The Moz Story

      Moz was the first & remains the most trusted SEO company.

    • Case Studies

      Explore how Moz drives ROI with a proven track record of success.

    • New Releases

      Get the scoop on the latest and greatest from Moz.

    Surface actionable competitive intel
    New Feature

    Surface actionable competitive intel

    Learn More
  • Log in
    • Moz Pro
    • Moz Local
    • Moz Local Dashboard
    • Moz API
    • Moz API Dashboard
    • Moz Academy
  • Avatar
    • Moz Home
    • Notifications
    • Account & Billing
    • Manage Users
    • Community Profile
    • My Q&A
    • My Videos
    • Log Out

The Moz Q&A Forum

  • Forum
  • Questions
  • Users
  • Ask the Community

Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

  1. Home
  2. Moz Tools
  3. Moz Pro
  4. Robots.txt blocking Moz

Moz Q&A is closed.

After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.

Robots.txt blocking Moz

Moz Pro
2
6
1.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as question
Log in to reply
This topic has been deleted. Only users with question management privileges can see it.
  • ChristinaRadisic
    ChristinaRadisic last edited by Aug 3, 2017, 5:19 AM

    Moz are reporting the robots.txt file is blocking them from crawling one of our websites.

    But as far as we can see this file is exactly the same as the robots.txt files on other websites that Moz is crawling without problems.

    We have never come up against this before, even with this site.

    Our stats show Rogerbot attempting to crawl our site, but it receives a 404 error.

    Can anyone enlighten us to the problem please?

    http://www.wychwoodflooring.com

    -Christina

    1 Reply Last reply Reply Quote 0
    • ChristinaRadisic
      ChristinaRadisic last edited by Aug 4, 2017, 6:44 AM Aug 4, 2017, 6:44 AM

      Hi Nigel

      Neither, they use server side filtering.Regards- David

      1 Reply Last reply Reply Quote 0
      • Nigel_Carr
        Nigel_Carr @ChristinaRadisic last edited by Aug 4, 2017, 6:30 AM Aug 4, 2017, 6:30 AM

        Hi David

        That's great news!

        As a matter of interest, where did they block it? as it's not in the Robots.txt - was in in htaccess.txt?

        Regards

        Nigel

        1 Reply Last reply Reply Quote 0
        • ChristinaRadisic
          ChristinaRadisic @Nigel_Carr last edited by Aug 4, 2017, 6:04 AM Aug 4, 2017, 6:04 AM

          Nigel,Thanks for the reply, the cgi-bin folder is never used by any of my sites but I put this in just as a matter of course, the folder would normally contain old cgi scripts so would not usually affect the crawling of a robot in any case.The reason for the problem turns out that our host had blocked rogerbot along with several other malicious bots, they have now lifted this block and the site is able to be crawled.- David

          Nigel_Carr 1 Reply Last reply Aug 4, 2017, 6:30 AM Reply Quote 0
          • Nigel_Carr
            Nigel_Carr last edited by Aug 3, 2017, 2:16 PM Aug 3, 2017, 2:16 PM

            Hi Christina

            I don't know how your site is set up but I can see that for some reason you are blocking access to the cgi-bin

            If that directory contains files that execute php or other permissions then that may well be your problem. It's the only directory you are blocking and since I haven't seen other Robots.tx blocking it, then I would hazard a guess that this is the root of your problem.

            Robots.txt

            User-agent: *
            Disallow: /cgi-bin/
            
            Sitemap: http://www.wychwoodflooring.com/sitemap.xml
            

            Regards

            Nigel

            ChristinaRadisic 1 Reply Last reply Aug 4, 2017, 6:04 AM Reply Quote 0
            • ChristinaRadisic
              ChristinaRadisic last edited by Aug 3, 2017, 6:58 AM Aug 3, 2017, 6:58 AM

              Our hosting provider has banned Rogerbot as they see it as problematic!!!!

              They are a great hosting provider so this is going to be a difficult one.

              1 Reply Last reply Reply Quote 0
              • 1 / 1
              1 out of 6
              • First post
                1/6
                Last post

              Got a burning SEO question?

              Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.


              Start my free trial


              Browse Questions

              Explore more categories

              • Moz Tools

                Chat with the community about the Moz tools.

              • SEO Tactics

                Discuss the SEO process with fellow marketers

              • Community

                Discuss industry events, jobs, and news!

              • Digital Marketing

                Chat about tactics outside of SEO

              • Research & Trends

                Dive into research and trends in the search industry.

              • Support

                Connect on product support and feature requests.

              • See all categories

              Related Questions

              • mrezair

                Unsolved Need Moz SEO Wordpress Plugin With API

                moz da wordpress seo tools

                Re: Moz WordPress Plugin? Hi guys,
                I need some Moz SEO Wordpress Plugins For my website working with Moz API. I've already found Moz DA-PA Checker plugin Moz DA-PA Checker But Need SEO Plugins too. Any Suggestion will be appreciated.

                Moz Pro | Aug 22, 2024, 2:27 AM | mrezair
                0
              • gallowaywebteam

                Unsolved Does Moz Pro include Moz Local

                moz local moz pro

                My client has bought about six Moz Local accounts and are pleased with results. We have not yet used your Moz Pro program. The client might be interested in switching to the Moz Pro if those Moz Local accounts can be included into it. Please let me know as soon as possible. Thanks!

                Moz Pro | Jan 18, 2024, 3:02 PM | gallowaywebteam
                0
              • BB_NPG

                Unsolved Rogerbot blocked by cloudflare and not display full user agent string.

                roger-bot crawl error

                Hi, We're trying to get MOZ to crawl our site, but when we Create Your Campaign we get the error:
                Ooops. Our crawlers are unable to access that URL - please check to make sure it is correct. If the issue persists, check out this article for further help. robot.txt is fine and we actually see cloudflare is blocking it with block fight mode. We've added in some rules to allow rogerbot but these seem to be getting ignored. If we use a robot.txt test tool (https://technicalseo.com/tools/robots-txt/) with rogerbot as the user agent this get through fine and we can see our rule has allowed it. When viewing the cloudflare activity log (attached) it seems the Create Your Campaign is trying to crawl the site with the user agent as simply set as rogerbot 1.2 but the robot.txt testing tool uses the full user agent string rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) albeit it's version 1.0. So seems as if cloudflare doesn't like the simple user agent. So is it correct the when MOZ is trying to crawl the site it uses the simple string of just rogerbot 1.2 now ? Thanks
                Ben Cloudflare activity log, showing differences in user agent strings
                2022-07-01_13-05-59.png

                Moz Pro | Jul 1, 2022, 12:14 PM | BB_NPG
                0
              • A_Q

                Robots.txt file issues on Shopify server

                We have repeated issues with one of our ecommerce sites not being crawled. We receive the following message: Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide. Are you aware of an issue with robots.txt on the Shopify servers? It is happening at least twice a month so it is quite an issue.

                Moz Pro | Oct 12, 2020, 12:27 PM | A_Q
                0
              • seochris17

                How long does Moz takes to update DA and PA value?

                I used to work daily on my office website https://quintdaily.com/ and  follow link building strategy. however, since more than 40-50 days, there is no movement in my DA, as the PA has just lifted up. However, there is no changes in Domain Authority so far. Besides, other than this website, im working on location wise SEO projects, but, i can find those websites Domain Authority changed in less 20 days. Is there is time period in changing the Domain Authority once it touched DA 30+?

                Moz Pro | May 9, 2024, 6:34 AM | seochris17
                2
              • NichGunn

                Should I set blog category/tag pages as "noindex"? If so, how do I prevent "meta noindex" Moz crawl errors for those pages?

                From what I can tell, SEO experts recommend setting blog category and tag pages (ie. "http://site.com/blog/tag/some-product") as "noindex, follow" in order to keep the page quality of indexable pages high. However, I just received a slew of critical crawl warnings from Moz for having these pages set to "noindex." Should the pages be indexed? If not, why am I receiving critical crawl warnings from Moz and how do I prevent this?

                Moz Pro | Nov 22, 2017, 11:13 AM | NichGunn
                0
              • shankar333

                What is Linking C-Blocks

                Currently i am using MOZ pro tool under moz analyticls >> Moz Competitive Link Metrics >> history having a graph "Linking C-Blocks" Please help me understanding Linking C-Blocks, what is, How to build, how to define ...

                Moz Pro | Oct 30, 2015, 3:28 PM | shankar333
                5
              • LesleyPaone

                How you can manipulate your MOZ DA

                I have become frustrated at MOZ in the last few months, none of my backlinks have made it into the index. Old back links. Long story short, I figured out the issue and I figured out how anyone can manipulate their DA. I wrote a blog post about it here, http://blog.dh42.com/manipulate-moz/

                Moz Pro | Jan 3, 2014, 11:54 PM | LesleyPaone
                1

              Get started with Moz Pro!

              Unlock the power of advanced SEO tools and data-driven insights.

              Start my free trial
              Products
              • Moz Pro
              • Moz Local
              • Moz API
              • Moz Data
              • STAT
              • Product Updates
              Moz Solutions
              • SMB Solutions
              • Agency Solutions
              • Enterprise Solutions
              Free SEO Tools
              • Domain Authority Checker
              • Link Explorer
              • Keyword Explorer
              • Competitive Research
              • Brand Authority Checker
              • Local Citation Checker
              • MozBar Extension
              • MozCast
              Resources
              • Blog
              • SEO Learning Center
              • Help Hub
              • Beginner's Guide to SEO
              • How-to Guides
              • Moz Academy
              • API Docs
              About Moz
              • About
              • Team
              • Careers
              • Contact
              Why Moz
              • Case Studies
              • Testimonials
              Get Involved
              • Become an Affiliate
              • MozCon
              • Webinars
              • Practical Marketer Series
              • MozPod
              Connect with us

              Contact the Help team

              Join our newsletter
              Moz logo
              © 2021 - 2025 SEOMoz, Inc., a Ziff Davis company. All rights reserved. Moz is a registered trademark of SEOMoz, Inc.
              • Accessibility
              • Terms of Use
              • Privacy

              Looks like your connection to Moz was lost, please wait while we try to reconnect.